FreeBSD SMP Project
Last updated: 10 September, 2001
Project Goal
The FreeBSD SMP project, often referred to as SMPng (SMP next generation), is focused on implementing fine-grained SMP support for the FreeBSD 5.0 kernel (scheduled for November 2001). Due to FreeBSD's history, this is much like trying to fit a square peg into a round hole, and as such, the intermediate results aren't pretty in many ways. We are specifically not attempting to rewrite the kernel from scratch, nor are we on a crusade to fix all the architectural nits currently present in the kernel. In fact, we expect to leave a trail of architectural nits that will still be evident in many ways when FreeBSD 5.0 is released. This is a pragmatic project rather than a theoretical one; we need to have the kernel working and stable in under a year, so time restraints require that we be realistic about what to do when.
Project Plan
This web page contains information related to the effort to improve SMP support in FreeBSD. In general, this project uses what it can from the BSD/OS 5.0 development kernel, and re-implements what cannot be directly used due to divergence in the code bases.
As with any free software project, a detailed schedule is not possible. We expect to have significant performance and stability issues that need to be worked through over the first several months of the project, though every effort will be made to keep -current running as well as possible.
Resources and Links
Status
Following is an incomplete list of tasks.
| Task | Responsible | Last updated | Status |
|---|---|---|---|
| Convert the giant lock from spinning to blocking, add the scheduler lock, add per-CPU idle processes. | Matt Dillon | 25 June 2000 | Done |
| Port the BSD/OS locking primitives (i386). | Jake Burkholder | 3 July 2000 | Done |
| Implement heavy-weight interrupt threads (i386). | Greg Lehey | 3 August 2000 | Done |
| Rewrite the low level interrupt code (i386 UP). | Greg Lehey | 3 August 2000 | Done |
| Demonstrated reasonable stability (self-hosted buildworld) (i386 UP). | -smp developers | 12 August 2000 | Done |
| Port the BSD/OS locking primitives (alpha). | Doug Rabson | 24 August 2000 | Done |
| Stub out (disable) spl()s. | Greg Lehey | 30 August 2000 | Done |
| Port the BSD/OS ktr code. | Greg Lehey, John Baldwin | 30 August 2000 | Done |
| Rewrite the low level interrupt code (i386 SMP). | John Baldwin | 1 September 2000 | Done |
| Demonstrated reasonable stability (self-hosted buildworld) (i386 SMP). | -smp developers | 6 September 2000 | Done |
| Demonstrated reasonable stability (self-hosted buildworld) (alpha). | -smp developers | 6 September 2000 | Done |
| Make malloc and friends thread-safe. | Jason Evans | 10 September 2000 | Done |
| Implement msleep(), make tsleep() an msleep() wrapper. | Jake Burkholder | 11 September 2000 | Done |
| Make fxp driver thread-safe. | Chuck Paterson | 17 September 2000 | Done |
| Make mbuf's thread-safe. | Bosko Milekic | 29 September 2000 | Done |
| Lock manager re-work. | Jason Evans | 3 October 2000 | Done |
| Implement heavy-weight interrupt threads (alpha). | John Baldwin, Doug Rabson | 5 October 2000 | Done |
| Rewrite the low level interrupt code (alpha). | Doug Rabson, John Baldwin | 5 October 2000 | Done |
| Process accounting. | Tor Egge, John Baldwin | 5 October 2000 | Done |
| Make ethernet drivers thread-safe. | Bill Paul | 15 October 2000 | Done |
| Make the mutex headers mostly machine-independent. | John Baldwin | 20 October 2000 | Done |
| Rename SMP_DEBUG to MUTEX_DEBUG. | John Baldwin | 20 October 2000 | Done |
| Give each soft interrupt its own thread. | Chuck Paterson | 25 October 2000 | Done |
| Make sf_bufs (sendfile(2)) thread-safe. | Bosko Milekic | 5 November 2000 | Done |
| Make the witness code work correctly. | John Baldwin | 18 November 2000 | Done |
| Protect network interface queues. | Jonathan Lemon | 24 November 2000 | Done |
| Split the ktr-specific code out of db_interface.c. | John Baldwin | 15 December 2000 | Done |
| Convert the sio driver to using a spin mutex. | John Baldwin | 18 December 2000 | Done |
| Implement condition variables. | Jake Burkholder, Jason Evans | 15 January 2001 | Done |
| Add a flag to mtx_init() (MTX_RECURSE) that denotes whether a mutex is allowed to recurse. | Bosko Milekic | 19 January 2001 | Done |
| Make the zone allocator thread-safe. | Dag-Erling Smorgrav | 21 January 2001 | Done |
| Convert simplelocks to mutexes. | Jason Evans | 24 January 2001 | Done |
| Make kernel preemptive with respect to interrupts. | Jake Burkholder | 31 January 2001 | Done |
| Cleanup of mutex API. | Bosko Milekic | 8 February 2001 | Done |
| Remove COM_LOCK. | Mark Murray | 11 February 2001 | Done |
| Merge various scheduling classes into one run queue. Modify scheduler to support preemptable kernel. | Jake Burkholder | 11 February 2001 | Done |
| Make priority propagation work correctly. | Jake Burkholder | 11 February 2001 | Done |
| Make most of the interrupt thread code MI and shared between hardware and software interrupts. | John Baldwin | 18 February 2001 | Done |
| Implement sx (shared/exclusive) locks. | Jason Evans | 5 March 2001 | Done |
| Generalize/improve witness to handle more complex locking primitives (mtx, sx). | John Baldwin | 28 March 2001 | Done |
| Convert the allproc and proctree locks from lockmgr locks to sx locks. | John Baldwin | 28 March 2001 | Done |
| Remove <sys/mutex.h> includes from other kernel headers such as <vm/vm_zone.h>, <sys/resourcevar.h>, <sys/ucred.h>, and <sys/mbuf.h>. | Mark Murray | 15 May 2001 | Done |
| Cleanup the various mp_machdep.c's, unify various SMP API's such as IPI delivery, etc. | John Baldwin | 15 May 2001 | Done |
| Make most of the forward_* and forwarded_* functions MI. | John Baldwin | 15 May 2001 | Done |
| Complete the MD support for SMP on the Alpha platform. | Andrew Gallatin, Doug Rabson, John Baldwin | 15 May 2001 | Done |
| Convert select() to use condition variables. | Seigo Tanimura | 15 May 2001 | Done |
| Add a "giant" lock around the VM subsystem. | Alfred Perlstein | 13 June 2001 | Done |
| Add a witness_assert() function to handle lock assertions. | John Baldwin | 27 June 2001 | Done |
| Extend sx locks to support try lock operations. | John Baldwin | 27 June 2001 | Done |
| Document KTR. | John Baldwin | 28 June 2001 | Done |
| Make fork_return, fork_exit, ast, and userret MI. | John Baldwin | 29 June 2001 | Done |
| Make sched_lock's savecrit a per-process property saved and restored in mi_switch and initialized in fork_exit. | John Baldwin | 30 June 2001 | Done |
| Make ast() loop. | John Baldwin | 10 August 2001 | Done |
| Add upgrade/downgrade sx lock operations. | Alexander Kabaev, Jason Evans | 13 August 2001 | Done |
| Implement semaphores. | Jason Evans | 14 August 2001 | Done |
| Add support for upgrade/downgrades in witness. | John Baldwin | 23 August 2001 | Done |
| Remove the MP safe syscall flag from the syscall table and add explicit mtx_lock/unlock's of Giant to all syscalls. | Matt Dillon | 7 September 2001 | Done |
| Make most of cpu_wait() and cpu_exit() MI. | Peter Wemm | 9 September 2001 | Done |
| Lock struct proc. | John Baldwin | 20 February 2001 | In progress |
| Make mbuf system use condition variables instead of msleep()/wakeup(). | Bosko Milekic | 21 March 2001 | In progress |
| Lock struct filedesc and struct file. | Seigo Tanimura, Alfred Perlstein | 14 May 2001 | In progress |
| Lock struct pgrp, struct session, and struct sigio. | Seigo Tanimura | 13 June 2001 | In progress |
| Split NFS into client and server. | Peter Wemm | 03 July 2001 | In progress |
| Make KTRACE write into tracefiles asynchronously. | John Baldwin | 6 September 2001 | In progress |
| Make the kernel fully preemptive. | John Baldwin | 7 September 2001 | In progress |
| Multithread the networking stack. | Alfred Perlstein, Jonathan Lemon Bosko Milekic | 7 September 2001 | In progress |
| Create mechanism in cdevsw structure to protect thread-unsafe drivers. | John Baldwin | 15 May 2001 | Stalled |
| Make printf() safe to call in almost any situation to avoid deadlocks. | Chuck Paterson | 15 May 2001 | Stalled |
| Implement trap handler for cmpxchgl on 80386 to support generic userland atomic operations. | Jake Burkholder | 15 May 2001 | Stalled |
| Add locking to NFS. | 15 May 2001 | Not Started | |
| Use per-CPU buffers for ktr to reduce synchronization. | 5 September 2000 | Not Started | |
| Remove priority argument from tsleep(), msleep(), cv_*wait*(). | 12 January 2001 | Not Started | |
| Implement lazy interrupt thread switching (context stealing). | 12 January 2001 | Not Started | |
| Lock struct ifnet. | 19 January 2001 | Not Started | |
| Reimplement kqueue using condition variables. | Jonathan Lemon | 15 March 2001 | Not Started |
| Conditionalize atomic ops in the SMP code that are used for debugging statistics. | Peter Wemm | 15 March 2001 | Not Started |
| Convert vm_map locks to sx locks. | John Baldwin | 13 June 2001 | Not Started |
| Add a new witness check for exiting processes to verify that an exiting process holds no locks. | John Baldwin | 13 June 2001 | Not Started |
| Make cpu_core MI. | 13 June 2001 | Not Started | |
| Lock struct pargs. | 20 June 2001 | Not Started | |
| Specify priorities for condition variables, semaphores, and sx locks. | 7 September 2001 | Not Started | |
| Fix SIGXPU and other #if 0'd things in mi_switch(). | 7 September 2001 | Not Started | |
| Fix synchronization of TLB flushes and invlpg() on x86 SMP. | 7 September 2001 | Not Started | |
| Axe schedpu() in favor of event driven priority updates as much as possible. | 7 September 2001 | Not Started | |
| Fix PHOLD() so that it blocks to guarantee PS_INMEM. | 7 September 2001 | Not Started | |
| Make grow_stack() MI. Possibly even a macro or inline. | 7 September 2001 | Not Started | |
| Fix *hold (e.g. crhold) to return reference to object. | 7 September 2001 | Not Started | |
| Fix various procfs_machdep.c to use PHOLD, not sched_lock. | 7 September 2001 | Not Started | |
| Convert eventhandlers to sx locks. | 7 September 2001 | Not Started | |
| Add witness checking for lockmgr locks. | 7 September 2001 | Not Started |
Known Issues
| Issue | Last updated | Status |
|---|---|---|
| Idle processor time is not charged to the idle processes. | 20 September 2000 | Resolved |
| microuptime creeps backwards. | 4 October 2000 | Resolved |
| microuptime() went backwards | 4 October 2000 | Resolved |
| Process accounting is not accurate (the more CPUs, the closer to correct it is). | 5 October 2000 | Resolved |
| M_DEVBUF is probably the wrong memory pool for interrupt stuff and we should think about creating a new malloc pool for that stuff. | 9 February 2001 | Resolved |
| PC card eject panics due to a race condition in the interrupt thread code. | 15 March 2001 | Resolved |
| SMP x86 boxes are seeing NCPU * 100 clk interrupts and NCPU * 128 rtc interrupts. | 15 May 2001 | Resolved |
| Witness will infinitely recurse when it acquires Giant after sleeping with a sleepable lock. | 27 June 2001 | Resolved |
| Serial gdb does not work if boot_ddb and boot_gdb options are specified. | 5 September 2000 | Unresolved |
| Serial gdb does not work at 115200 baud. | 5 September 2000 | Unresolved |
| Profiling is broken. | 20 February 2001 | Unresolved |
News
The remainder of this page is structured as a reverse-chronological log.
15 May 2001