Pushing FreeBSD's SMP performance to the next level

Pushing FreeBSD's SMP performance to the next level

FreeBSD developers are busy changing the kernel to allow fine-grained locking and improved SMP performance.

From: Jason Evans <jasone@freebsd.org>
FreeBSD SMP project manager

FreeBSD has had rudimentary SMP support since version 3.0, but until recently, there was no clear path toward better performance. SMP support in FreeBSD 3.x and 4.x uses what is commonly called a giant kernel lock. The concept of the giant lock is very simple: if running in the kernel, you must own the giant lock. This keeps more than one CPU from ever running in the kernel at any given time.

The disadvantages of such a scheme are clear. No matter how many CPUs there are, only one CPU worth of kernel work can be done. This is a serious enough bottleneck for many types of work loads (for example I/O-intensive applications) that the extra cost of SMP machines over uniprocessor machines running FreeBSD 3.x or 4.x often doesn't make sense.

Matt Dillon made some improvements in FreeBSD 4.x by moving certain portions of the system call code outside of the giant lock. This work made a measurable improvement to performance. However, something more radical was needed in order to push performance to an entirely new level.

Early this year, BSDi and Walnut Creek CD-ROM merged. The combined company has interests in both BSD/OS (BSDi's proprietary 4.4-lite-based operating system) and FreeBSD, and is interested in seeing cross pollination. So, BSDi made the BSD/OS source code available to FreeBSD developers and encouraged the developers to integrate portions of BSD/OS that could enhance FreeBSD.

One of the technologies that BSDi has been developing for BSD/OS 5.0 (BSDi is currently shipping BSD/OS 4.x) over the past couple of years is fine-grained SMP functionality. FreeBSD developers had been lacking clear direction in how to improve SMP support, so having access to BSD/OS 5.0's SMP implementation has been a serious boon. Not only have we been able to take portions of BSD/OS's code, but it has provided us with two even more important items: 1) A clear plan of how to move forward, and 2) very strong evidence that it will work, and work well.

In June a group of about 20 FreeBSD developers got together with Chuck Paterson, one of the main BSDi developers who has been working on SMP for BSD/OS 5.0. Chuck spent a day explaining how BSD/OS does things, and then we spent a day discussing how to apply that technology to FreeBSD. Between then and September 6th, a small group of developers quietly passed around patches as they ported portions of BSD/OS to FreeBSD and implemented the bare bone basics of BSD/OS's SMP architecture. On September 6th, the result of that work was committed to cvs as part of the mainline -current development tree. Any daring souls are invited to track development as it continues, though we all expect there to be some pretty rough road ahead for the next several months.

Thus far, this article has said little about how the new SMP code actually works. Following is a short conceptual summary.

Traditionally, the Unix kernel has been completely single-threaded, with the caveat that interrupts make things a tad more complex. In order to protect critical regions of code from interrupts, the spl*() (set priority level) functions were used. The new SMP architecture makes each interrupt handler a separate thread, so that interrupt handlers can be scheduled independently. This completely removes the need for the spl*() functions, since when an interrupt comes in, that interrupt is handled within the context of a thread devoted to the interrupt. This change allows interrupts to seamlessly fit within a normal multi-threading scheme. Now, instead of protecting against interrupts with spl*(), code can protect itself with mutual exclusion locks that are specific to particular portions of the kernel code. The possibilities for improved concurrency are very significant, and this is where we will be spending much of our effort, now that most of the ground work is laid.

If you are interested in details about the people involved in the SMP project, technical information, or current status, see http://people.freebsd.org/~jasone/smp/.