Linux STREAMS (LiS)

Kernel Compatibility Issues Concerning LiS

Kernel and Driver Compatibility

LiS-2.19 can be described as LiS-2.18 with further Linux 2.6 changes as well as a number of SMP-related hangs and a panic fixed

LiS-2.18 contains a number of changes from LiS-2.16. LiS-2.17 was pretty much an experimental version. Documented here are the principle changes between 2.16 and 2.18.

NOTE: STREAMS drivers must be recompiled against LiS-2.18 or later header files in order to utilize those LiS versions. Do NOT attempt to install LiS-2.18 "underneath" drivers compiled against LiS-2.16 or LiS-2.17.

Here are the principal changes. I am not documenting all the little bug fixes, just the main changes that users may find visible.

  • OpenSS7 inet driver in, then out, then in, then out of LiS distribution. Currently "out". Download separately from openss7.org.
  • Discontinue support for 2.2 kernels.
  • More sophisticated choosing of version of C compiler to use when building LiS.
  • Several iterations of timeout/untimeout implementation ending with a blend of the original, plus hash table for lookup speed, plus use of kernel cache blocks to allocate structures.
  • Organize so as not to hold any spin locks when entering STREAMS driver functions. This means that queue locking now uses a semaphore instead of a spin lock. This means, in turn that you really cannot call putnext() from interrupt context.
  • Some contributions for LiS on S/390, PPC, IA64 and SPARC.
  • Eliminate poll() routine from libLiS. You should be using the standard libc version.
  • Some contributed fixes for fattach, pipes and fifos.
  • Reworking of open routine for races involving open vs open and open vs close.
  • Eliminate use of special LiS system calls to kernel. Getpmsg uses the read function and putpmsg uses the write function.
  • Some contributions to make rpm building easier.
  • A contribution for performance enhancement mainly involving use of kernel caches.
  • More performance enhancements involving minimizing of lock contention, and an LiS mechanism to track lock contention.
  • Tracking code for queue contention to assist users in optimizing.
  • Code to build a histogram of semaphore wakeup latency and print out the table.
  • Compatibility with 2.6 kernel.
  • Revision of internal LiS definition of dev_t to incorporate 12 bits of major device number and 20 bits of minor device number.
  • A contribution for quieter makes with 2.6 style progress messages. Use "make V=1" for verbose make output.
  • A new file <sys/LiS/module.h> for loadable drivers to include as the first include file to get include file ordering involving <linux/module.h> correct portably for both 2.4 and 2.6 kernels.
  • Improved queue scheduling, which is responsible for more performance improvements, especially on multiple CPU systems.
  • A contribution to renumber message types for compatibility with other STREAMS implementations.
  • Queues are no longer run on user threads. They are only run from the LiS queue scheduler threads.
  • Add lis_down_nosig() function to do a "down" with signals blocked. Useful in close routines that must sleep.
  • Streams command option "-L" to print out lock contention tracking.
  • Qlock command for strconf to control queue locking style.
  • Qlock registration routine for loadable modules to specify their queue locking style.
  • Putnext() et al no longer free the message upon failure as per SVR4 specs.
  • Improved errno reporting in strtst.
  • Implement a message deferred list in each queue and use it for puts to queues that are in a "qprocsoff" state.
  • Implement freezestr() and defer message processing while stream is frozen.
  • Use of freezestr() in close and module pop code.
  • Greatly reduce the number of exported symbols.

Compatibility of LiS-2.18 with 2.6 Kernel

There are several issues that needed to be addressed for compatibility with the 2.6 Linux kernel. You are encouraged to follow the links in the paragraphs below to see more detailed information on each of these topics.

  1. The 2.6 kernel redefined the size of the dev_t structure. LiS has extended its internal dev_t structure to be compatible with the 2.6 method for some time. For more information click here.
  2. The 2.6 kernel changed the approach to building and installing loadable modules. This affects LiS as a whole and also affects how you install separate loadable STREAMS drivers. LiS provides a mechanism that allows STREAMS drivers and modules to be easily installed.
  3. The 2.6 kernel offers an option to compile the kernel using machine registers to pass parameters to functions. LiS takes this into account.
  4. The 2.6 kernel needs GCC version 3.3.3 to be compiled properly. LiS needs to be compiled using this same version of the compiler when running with the 2.6 kernel.
  5. You may have to edit the file /etc/rc.d/rc.sysinit in order to get demand loadable modules to work correctly. This is especially true when hosting a 2.6 kernel on a 2.4 distribution.

LiS-2.16 Kernel and Driver Compatibility

LiS-2.16 is a small change from LiS-2.15. The change is that it no longer uses Linux system calls to implement getpmsg and putpmsg. Instead it overloads the read and write file system functions with particular values for the count parameter, values that are otherwise invalid.

LiS-2.15 Kernel and Driver Compatibility

LiS-2.15 continues to insulate STREAMS drivers from the Linux kernel. It works with 2.2, 2.4, and 2.5 versions of the kernel. Support for 2.0 kernels has been dropped.

Driver writers will need to recompile their drivers against LiS-2.15 include files. You will see the following major changes.

  • LiS spin locks and semaphores have been rearranged so that the kernel memory is at the end of the structure instead of the beginning.
  • The former change allows for there to be dynamic allocation routines for spin locks and semaphores.
  • LiS now provides an abstraction for read/write locks, with dynamic allocation.
  • Those experimenting with 2.5 kernels will notice that the "sleep while holding spin lock" problems have been fixed.
  • Porting to 2.5 has necessitated some changes to the major/minor device structure handling.
  • The fattach related functions are functional on kernels version 2.4.7 and later.
  • STREAMS pipes and FIFOs are now functional.
  • OS interface code has been added for the kernel's DMA mapping functions.

There is one known bug in LiS-2.15 relative to 2.5 kernels. It has to do with a memory leak involving timer structures, and may prove to be a kernel bug rather than an LiS bug. Since the 2.5 kernel is not suitable for general use I am saving the investigation of this bug for later.

LiS-2.14 Kernel and Driver Compatibility

LiS-2.13 was a series of beta releases. LiS-2.14 represents the culmination of this series. There should be enough distribution and kernel compatibility that LiS-2.14 will hold up for some time.

The known fattach and FIFO bugs have still not been fixed. The author of those subsystems has not found the time to put in the fixes, nor have I.

LiS-2.13 Kernel and Driver Compatibility

This version of LiS has been tested with 2.4 kernels up to 2.4.16. LiS does not yet support the fattach/fdetach functions on kernel versions 2.4.7 and beyond. There are also known bugs in the LiS pipe/FIFO code. All of these problems are scheduled to be fixed in early 2002.

LiS-2.13 adds the ability for drivers to make their own "/dev" nodes via the lis_mknod function. Also provided is an lis_unlink function that allows drivers to remove their device files.

There is almost no new functionality added by LiS-2.13. The differences between LiS-2.13 and LiS-2.12 are almost entirely kernel compatibility issues and bug fixes.

LiS-2.12 Kernel and Driver Compatibility

This version of LiS is compatible with all 2.2.x versions of the kernel and with early versions of the 2.4.x kernel, at least up to 2.4.2 and perhaps later versions as well.

If you have drivers that have worked with LiS-2.10 or LiS-2.11 (or earlier) please recompile them using the header files from LiS-2.12. This may be the last recompile in quite some time that you will need for your driver code.

LiS-2.12 contains a sufficient Driver/Kernel Interface (DKI) that it is straightforward to write a STREAMS driver that can be compiled against LiS-2.12 and the resulting object modules used either on a 2.2 or 2.4 kernel, with only LiS needing recompilation on the target machine.

When run on 2.4 kernels, LiS makes full use of multiple CPUs. It forks a queue runner task for each CPU and locks each task onto its CPU. Queue runner tasks are awakened to assist with service procedure processing as the number of scheduled queues increases.

Because of this aggressive use of processors, you may find that your drivers do not function properly when run with LiS-2.12 in a multi-CPU SMP environment. You should expect that drivers that worked in single-CPU environments will continue to work as before.

Making your drivers MP safe involves the use of spin locks. The DKI documentation contains advice on the use of these locks.

This version of LiS also contains a rewrite of the flushing code and tests added to strtst for flushing. In particular the details of the rules for flushing queue bands are now adhered to. Be advised, however, that Solaris STREAMS does not adhere strictly to these rules so there may be some subtle differences in behavior between LiS and Solaris when flushing queue bands.

Speaking of queue bands, the queue band handling code has been debugged a bit more and a test added to strtst to illustrate its correct behavior.

Differences between LiS-2.12.2 and LiS-2.12.1

  • LiS installation will attempt a kernel "make dep" if modversions.h is needed but absent

Differences between LiS-2.12.1 and LiS-2.12

  • Added an environment variable to control the base of the LiS major device numbers.
  • LDL needed ifdef for ETH_P_ECHO for 2.4.4 (de-implemented symbol).
  • Queue runner threads change to root directory at startup time.
  • Fixed a lock ordering problem with lis_backenable.
  • Stopped acquiring the inode i_sem; not needed and led to lock ordering problems.

Differences between LiS-2.12 and LiS-2.11

The following is a list of differences between LiS-2.11 and LiS-2.12. The list may not be complete.

  • The aggressive use of multiple CPUs on 2.4 kernels is the biggest difference.
  • Debugging support for spin locks and semaphores.
  • Rewrite of flushing code to adhere to AT&T semantics for flushing queue bands.
  • Improve queue band handling code.
  • Fix some problems with orphan inode structures with pipes, fifos and fattach.
  • Put and service procedure prototypes changed to return int rather than void (AT&T compatibility).
  • Propogate task credentials (user id and capabilities) of stream opener to service procedures.
  • Ensure that opener's user id is associated with the opened stream.
  • Use defined type toid_t for return from timeout function.
  • M_SIG causes signal as soon as it reaches front of queue without having to be read.
  • Fix problem with flushing stream head queue with M_PCPROTO at front.
  • Getpmsg did not always update the strbuf len field correctly for empty messages.
  • Cross compile patches for PPC from Wolfgang Denk.
  • Change in LiS major device numbers.
  • Support for 2.0 kernels no longer maintained. If you are using an older kernel you must use an older LiS version as well (try LiS-2.8).
  • No longer use /etc/ld.so.preload. Instead just copy LiS libs to /lib and run ldconfig.
  • Make very-clean now removes LiS libs as well as other stuff.

LiS-2.10 Kernel and Driver Compatibility

This version of LiS is compatible with all 2.2.x versions of the Linux kernel. It may work with 2.4.x kernels, but you should probably wait for LiS-2.11 for that.

If you have drivers that worked with LiS-2.8 or earlier, you must recompile your drivers in the context of the LiS-2.10 header files. The queue_t structure has changed in size since LiS-2.8 which means that the old RD and WR macros will not compute the correct addresses.

LiS-2.10 contains features that are intended to greately reduce the necessity of recompiling STREAMS driver code in future versions of LiS or future versions of the kernel. The goal is to be able to compile STREAMS drivers against LiS-2.10 header files and use the resulting object code on both 2.2.x kernels and 2.4.x kernels.

For more details about the interface between STREAMS drivers and the kernel, see the Driver/Kernel Interface documentation.

New Features in LiS-2.11

LiS-2.11 only differs from LiS-2.10 in that LiS-2.11 runs on kernel version 2.4.0 in addition to the 2.2 series of kernels.

NOTE: When running with a 2.4 kernel, make sure that you are using modutils at least as recent as 2.3.21. The alignment of the text segment in loadable modules changed in the 2.4 kernel and the old modutils do not understand the new alignment. The strtst program will not run correctly unless LiS is loaded using the newer modutils. You can find the source for the new modutils at http://www.kernel.org/pub/linux/utils/kernel/modutils/v2.3.

New Features in LiS-2.10

LiS-2.10 contains a number of new features since LiS-2.8. LiS-2.9 was largely a beta release that tested these features.

  • fattach/fdetach - These SVR4 functions have finally been implemented.
  • STREAMS pipes - The default LiS library contains a "pipe" routine that uses STREAMS to implement inter-process pipes.
  • STREAMS fifos - You can now write pipe server processes to which clients can attach simply by opening a named file.
  • SMP Safety - LiS will now run on multi-CPU systems.
  • Driver insulation - LiS now provides a rich set of interface routines that insulate a STREAMS driver from the kernel version on which it is running.
  • Improved Configuration - The installation script recognizes more versions and distributions of the kernel.
  • libc Compatibility - LiS and glibc are getting more compatible all the time. The LiS and glibc stropts.h files are now completely compatible.
  • Improved Documentation - There is at least some documentation for all of the new features.
  • Searchable Documentation - Gcom has published a new web site at http://www.gcom.com with search capabilities. This includes the ability to search LiS documentation from the web.

Kernel Version 2.3.x

For LiS version 2.7 and later and for kernel version 2.3.x there are some significant compatibility issues.  Click here for more on this topic.

Kernel Version 2.2.x

For LiS version 2.5 and later and for kernel version 2.2.x there are no compatibility issues; there are no kernel patches whatsoever required to install LiS. You will need LiS-2.4 at minimum to run in a 2.2.x kernel.

Kernel Version 2.0.36

The latest version of LiS has not been tested on 2.0 kernels. Therefore, do not be surprised if it does not install or execute correctly in these kernels. If you are using an old kernel, you must also use an older version of LiS, perhaps LiS-2.5.

For LiS version 2.5 and later and for kernel version 2.0.36 there are no kernel patches required to run LiS as a "bottom half" process. A one-line patch is required to run LiS as a kernel daemon process. The installation default is to run as a bottom half process in 2.0.36. LiS-1.25 or later should install properly with 2.0.36. The more recent the version of LiS, the less kernel patching is required.

Extracting Old LiS Kernel Patches (obsolete)

Previous versions of LiS installed inside the kernel source tree and included patches to the kernel in order to install LiS into the kernel. There was no provision made for removing these patches from the kernel. 

Beginning with LiS-1.25 any patches made to kernels whose version number is 2.0.35 or later can be removed from the kernel just by doing a "make realclean" in the LiS installation directory. If you have installed the older version of LiS in your kernel then you need to take steps to remove it prior to installing this version of LiS. 

There are two methods of doing this. Both are manual procedures. 

  1. Change the name of your kernel source directory, as in
    2.       mv /usr/src/linux /usr/src/linux.old
    and then untar a fresh kernel source as /usr/src/linux. This method works fine if LiS was the only patch to your kernel. Once the new kernel works, remove the linux.old directory. 
  2. In your kernel tree, do the following:
     cd /usr/src/linux/drivers

     rm -r streams
That deletes almost all of the files that came with the old version of LiS. 

Next, you must restore the patched kernel files to their original state. From the directory /usr/src/linux, look for the following files: 

LiS: 

    • arch/i386/kernel/entry.S.orig 
    • fs/read_write.c.orig 
    • fs/select.c.orig 
    • kernel/ksyms.c.orig 
    • init/main.c.orig 
    • include/asm-i386/unistd.h.orig 
    • drivers/Makefile.orig
  • Semaphores: 
    •
      • arch/i386/lib/semaphore.S.orig 
      • kernel/sched.c.orig 
      • include/asm/semaphore.h.orig
  • If you do not have any other patches applied to these files, then change their names back to the original file names (these names minus the .orig suffix). Not all of these files will be present on every version of the kernel. Kernels later than 2.0.30 would not have had the semaphore patch, for example. 

    •  

     

    If you have other patches in these files then you will need to consult one of the following files from the newly installed /usr/src/LiS directory: 

      • patches.kernel.24-27 
      • patches.kernel.28-34
  • There is really no recourse at this point other than to examine the patches and manually extract the LiS patches from the affected files. 
    •

    Open Flags (obsolete)

    The open flags MODOPEN and CLONEOPEN have been changed since the previous version. You must recompile your STREAMS drivers using the new header files in order to incorporate this change. The change brings the bit assignments into conformance with SVR4. 

    STREAMS drivers compiled against the old header files should not be linked into LiS without recompiling. The old open routines will misinterpret their flags argument. 

    System Calls (obsolete)

    This change affects all application level programs that use the STREAMS constructs getpmsg, putpmsg or poll. 

    For kernel versions prior to 2.0.36, LiS did not have official system call numbers assigned for getpmsg, putpmsg and poll. Beginning with kernel version 2.0.36, and all 2.2 kernels, LiS has official system call numbers assigned for getpmsg and putpmsg. The 2.2 kernels have a built-in poll system call. Therefore, there is no LiS poll system call in 2.2 kernels. 

    The system call slots that LiS used for getpmsg and putpmsg in earlier kernels were, of course, taken up by other functions by the time the official system call slots got assigned for the 2.2 (and 2.1) kernel. That means that older STREAMS applications compiled for use with earlier versions of LiS, when run on newer kernels, will be issuing incorrect system calls for getpmsg and putpmsg. These applications need to be recompiled and relinked using the new LiS. 

    The new version of LiS, even when run on kernels prior to 2.0.36, will always plug the system call table slots for the new "official" STREAMS system calls in addition to the older ad hoc slots. This means that you can use applications that utilize putpmsg and getpmsg with the new system calls on older kernels by using the new version of LiS. 

    However, the poll system call is not so easy. If your application uses poll then you must maintain a version of it that is compatible with kernel versions prior to 2.0.36 and another version that is compatible with kernel versions 2.0.36 and beyond, including 2.2 kernels. The reason for this is that prior to 2.0.36, LiS contained the only implementation of the poll system call. However, starting with 2.0.36, the kernel contains its own implementation of poll and the LiS implementation must yield to the kernel's implementation. The system call slot used by LiS for poll and the one used by the Linux kernel are different. So there is really no way around maintaining two versions of such applications. 

    The following table summarizes the system call number assignments for different kernel versions. Numbers in plain text are the ad hoc numbers used by earlier versions of LiS. Numbers in italics are official kernel assigned numbers. The problem with the poll system call is that the number 169, used by earlier version of LiS, is used for a different system call in kernel version 2.0.36 and beyond. Thus, it is not possible for LiS to plug that system call slot with a pointer to its poll routine for backward compatibility. 
     
    Version Getpmsg Putpmsg Poll
    LiS for kernel versions < 2.0.36 168 167 169
    LiS for kernel version 2.0.36 188 189 168
    LiS for kernel versions >= 2.2.0 188 189 168

    In kernel versions 2.0.36 and earlier, LiS provides the implementation of the poll system call. In 2.2 kernels, the kernel provides the implementation of poll and LiS hooks into it on behalf of its STREAMS drivers. 

    Libraries 

    The older version of LiS had direct inline code for STREAMS system calls defined in stropts.h. The new version contains function prototypes in stropts.h with the actual system call code contained in a library that you link with your application program. 

    The library resides in the directory /lib and is named libLiS.a. Thus, you need to include the directive "-lLiS" with the link of your STREAMS applications. For more about LiS libraries click here. 

    Interactions with Other Packages 

    In kernel versions around 2.2.14 there were some compile time clashes with the irda driver. These have been resolved and/or worked around in LiS-2.10.

    On kernel version prior to 2.0.36 there can be problems with LiS interacting with other packages such as JDK and, perhaps, iBSC. In particular, JDK looks to see if system call number 168 has been assigned. If it has, it assumes that the operating system implements the poll system call in that slot. As you can see from the chart above, older versions of LiS did plug system call 168, but with the getpmsg routine, not with poll. Although we do not know for sure there is a possibility that iBSC would have the same problem. 

    The solution is to upgrade to a kernel at least as new as 2.0.36 or a 2.2 series kernel. 

    Compatibility with Kernel Version 2.3.x

    Version 2.3 of the Linux kernel brings with it some compatibility issues that need to be addressed by the LiS user.  The two most important ones concern the file <sys/stropts.h> and the major device numbers used by LiS.

    stropts.h Compatibility

    There are no more compatibility problems with <sys/stropts.h> with glibc-2.1 and LiS-2.10. The following is more for historical purposes than practical necessity.

    Beginning at least with egcs-2.91.66 (egcs-1.1.2 release), which comes with Red Hat 6.0, there is a file in the standard include directory named <sys/stropts.h>.  This file has constant definitions that are incompatible with those used in  LiS/include/sys/stropts.h.  If you compile an application against the glibc version of stropts.h, and compile LiS using its own version then certain ioctls may not work correctly.  You should be aware of this problem and be sure to include "-I/usr/src/LiS/include" in the compiler options that you use in compiling your STREAMS based applications.

    In this version of LiS, some of the constants in stropts.h have been changed to conform to the values used by UnixWare and Solaris.  These are different values than previously used in LiS.  When you install LiS the installation procedure will ask you whether you want LiS compiled with the backward-compatible LiS constants, or the UnixWare/Solaris compatible constants.  Logically speaking, it does not matter which set you use as long as LiS and your application code are both compiled with the same values.

    I highly recommend that you use the UnixWare/Solaris compatible version, however.  A future release of egcs, utilizing glibc 2.2, will contain an updated version of its stropts.h which has constants that are compatible with UnixWare, Solaris and LiS.  So by selecting the UnixWare/Solaris compatible version at this time you can ensure that your applications will be fully compatible with these values in the future.

    With any luck, these constants will never have to change again.

    Major Device Number Compatibility

    The second major compatibility issue concerns the major device numbers that LiS assigns to STREAMS devices.  In the past LiS based these device numbers at 50, since the Linux kernel did not pre-define many major device numbers.  As of kernel version 2.3.x there are major device numbers defined up to 220 and beyone!  So starting with LiS-2.12, we have used the major number of 240 as the base for STREAMS device files. This range is supposed to be reserved for "experimental drivers" which should make it safe to use.

    What this means is that you must be sure to run the strmakenodes program before running any STREAMS applications after installing LiS-2.12.  This need not concern you overly, since doing a "make install" in the /usr/src/LiS directory causes strmakenodes to be run anyway.  This is more a concern if you are compiling LiS on one machine and then loading it onto another for execution.  In such cases you may need to load the new strmakenodes program and run it.

    I am hoping that the kernel developers will expand the major and minor device number spaces for 2.6. If they do that then LiS should be able to get a block of majors allocated to it.