FreeBSD SMP Project

FreeBSD SMP Project

Last updated: 22 February 2001.

Project Goal

The FreeBSD SMP project, often referred to as SMPng (SMP next generation) is focused on implementing fine-grained SMP support for the FreeBSD 5.0 kernel (scheduled for mid-2001). Due to FreeBSD's history, this is much like trying to fit a square peg into a round hole, and as such, the intermediate results aren't pretty in many ways. We are specifically not attempting to rewrite the kernel from scratch, nor are we on a crusade to fix all the architectural nits currently present in the kernel. In fact, we expect to leave a trail of architectural nits that will still be evident in many ways when FreeBSD 5.0 is released. This is a pragmatic project rather than a theoretical one; we need to have the kernel working and stable in under a year, so time restraints require that we be realistic about what to do when.

Project Plan

This web page contains information related to the effort to improve SMP support in FreeBSD. In general, this project uses what it can from the BSD/OS 5.0 development kernel, and re-implements what cannot be directly used due to divergence in the code bases.

As with any free software project, a detailed schedule is not possible. We expect to have significant performance and stability issues that need to be worked through over the first several months of the project, though every effort will be made to keep -current running as well as possible.

Resources and Links

SMP kernel synchronization rules [http://people.freebsd.org/~jasone/smp/smp_synch_rules.html].
Most SMP-related discussion takes place on the freebsd-smp mailing list. Follow this link [http://www.freebsd.org/handbook/eresources.html] to a portion of the FreeBSD Handbook for mailing list details.
Steve Passe has been maintaining a SMP project page [http://people.freebsd.org/~fsmp/SMP/SMP.html] that contains additional information, and goes back further in time than this web page.
Matt Dillon is maintaining a web page [http://apollo.backplane.com/FreeBSDSmp/] that documents the work he is doing on the SMP project.

Status

Following is an incomplete list of tasks. If you know of tasks that you think belong on this list, or if there are inaccuracies, please send additions/changes to <jasone@freebsd.org>.

Task	Responsible	Last updated	Status
Convert the giant lock from spinning to blocking, add the scheduler lock, add per-CPU idle processes.	Matt Dillon	25 June 2000	Done.
Port the BSD/OS locking primitives (i386).	Jake Burkholder	3 July 2000	Done.
Implement heavy-weight interrupt threads (i386).	Greg Lehey	3 August 2000	Done.
Rewrite the low level interrupt code (i386 UP).	Greg Lehey	3 August 2000	Done.
Demonstrated reasonable stability (self-hosted buildworld) (i386 UP).	-smp developers	12 August 2000	Done.
Port the BSD/OS locking primitives (alpha).	Doug Rabson	24 August 2000	Done.
Stub out (disable) spl()s.	Greg Lehey	30 August 2000	Done.
Port the BSD/OS ktr code.	Greg Lehey, John Baldwin	30 August 2000	Done.
Rewrite the low level interrupt code (i386 SMP).	John Baldwin	1 September 2000	Done.
Demonstrated reasonable stability (self-hosted buildworld) (i386 SMP).	-smp developers	6 September 2000	Done.
Demonstrated reasonable stability (self-hosted buildworld) (alpha).	-smp developers	6 September 2000	Done.
Make malloc and friends thread-safe.	Jason Evans	10 September 2000	Done.
Implement msleep(), make tsleep() an msleep() wrapper.	Jake Burkholder	11 September 2000	Done.
Make fxp driver thread-safe.	Chuck Paterson	17 September 2000	Done.
Make mbuf's thread-safe.	Bosko Milekic	29 September 2000	Done.
Lock manager re-work.	Jason Evans	3 October 2000	Done.
Implement heavy-weight interrupt threads (alpha).	John Baldwin, Doug Rabson	5 October 2000	Done.
Rewrite the low level interrupt code (alpha).	Doug Rabson, John Baldwin	5 October 2000	Done.
Process accounting.	Tor Egge, John Baldwin	5 October 2000	Done.
Make ethernet drivers thread-safe.	Bill Paul	15 October 2000	Done.
Make the mutex headers mostly machine-independent.	John Baldwin	20 October 2000	Done.
Rename SMP_DEBUG to MUTEX_DEBUG.	John Baldwin	20 October 2000	Done.
Give each soft interrupt its own thread.	Chuck Paterson	25 October 2000	Done.
Make sf_bufs (sendfile(2)) thread-safe.	Bosko Milekic	5 November 2000	Done.
Make the witness code work correctly.	John Baldwin	18 November 2000	Done.
Protect network interface queues.	Jonathan Lemon	24 November 2000	Done.
Split the ktr-specific code out of db_interface.c.	John Baldwin	15 December 2000	Done.
Convert the sio driver to using a spin mutex.	John Baldwin	18 December 2000	Done.
Implement condition variables.	Jake Burkholder, Jason Evans	15 January 2001	Done.
Add a flag to mtx_init() (MTX_RECURSE?) that denotes whether a mutex is allowed to recurse.	Bosko Milekic	19 January 2001	Done.
Make the zone allocator thread-safe.	Dag-Erling Smorgrav	21 January 2001	Done.
Convert simplelocks to mutexes.	Jason Evans	24 January 2001	Done.
Make kernel preemptive.	Jake Burkholder	31 January 2001	Done.
Cleanup of mutex API.	Bosko Milekic	8 February 2001	Done.
Remove COM_LOCK.	Mark Murray	11 February 2001	Done.
Merge various scheduling classes into one run queue. Modify scheduler to support preemptable kernel.	Jake Burkholder	11 February 2001	Done.
Make priority propagation work correctly.	Jake Burkholder	11 February 2001	Done.
Make most of the interrupt thread code MI and shared between hardware and software interrupts.	John Baldwin	18 February 2001	Done.
Split NFS into client and server.	Peter Wemm	20 September 2000	In progress.
Make the IP stack thread-safe.	Alfred Perlstein	8 September 2000	In progress.
Create mechanism in cdevsw structure to protect thread-unsafe drivers.	John Baldwin	5 October 2000	In progress.
Implement trap handler for cmpxchgl on 80386 to support generic userland atomic operations.	Jake Burkholder, Jason Evans	9 December 2000	In progress.
Implement sx (shared/exclusive) locks.	Jason Evans	11 January 2001	In progress.
Lock struct proc.	John Baldwin	20 February 2001	In progress.
Make fork_return, fork_exit, ast, and userret MI.	John Baldwin	22 February 2001	In progress.
Cleanup the various mp_machdep.c's, unify various SMP API's such as IPI delivery, etc.	John Baldwin	19 January 2001	In progress.
Make printf() safe to call in almost any situation to avoid deadlocks.	Chuck Paterson	25 January 2001	In progress.
Add locking to NFS.	Peter Wemm.	20 September 2000	Not started.
Use per-CPU buffers for ktr to reduce synchronization.		5 September 2000	Not started.
Remove priority argument from tsleep(), msleep(), cv_wait().		12 January 2001	Not started.
Implement lazy interrupt thread switching (context stealing).		12 January 2001	Not started.
Make mbuf system use condition variables instead of msleep()/wakeup().	Bosko Milekic	16 January 2001	Not started.
Lock struct filedesc.		19 January 2001	Not started.
Lock struct pgrp.		19 January 2001	Not started.
Lock struct sigio.		19 January 2001	Not started.
Lock struct session.		19 January 2001	Not started.
Lock struct ifnet.		19 January 2001	Not started.
Make most of the forward_* and forwarded_* functions MI.	John Baldwin	19 January 2001	Not started.
Document KTR.	John Baldwin	19 January 2001	Not started.
Make the VM thread-safe.	Dag-Erling Smorgrav	22 January 2001	Not started.
Generalize/improve witness to handle more complex locking primitives (mtx, sx).	John Baldwin, Jason Evans	6 February 2001	Not started.
Convert select() to use condition variables.		25 January 2001	Not started.
Add an MPSAFE flag to all syscall lists.		6 February 2001	Not started.

Known Issues

Issue	Last updated	Status
Idle processor time is not charged to the idle processes.	20 September 2000	Resolved.
microuptime creeps backwards.	4 October 2000	Resolved.
microuptime() went backwards	4 October 2000	Resolved.
Process accounting is not accurate (the more CPUs, the closer to correct it is).	5 October 2000	Resolved.
M_DEVBUF is probably the wrong memory pool for interrupt stuff and we should think about creating a new malloc pool for that stuff.	9 February 2001	Resolved.
PC card eject panics due to a race condition in the interrupt thread code.	20 February 2001	In progress.
Serial gdb does not work if boot_ddb and boot_gdb options are specified.	5 September 2000	Unresolved.
Serial gdb does not work at 115200 baud.	5 September 2000	Unresolved.
Profiling is broken.	20 February 2001	Unresolved.

News

The remainder of this page is structured as a reverse-chronological log.

24 January 2001

A status report [http://docs.freebsd.org/cgi/getmsg.cgi?fetch=38125+0+archive/2001/freebsd-smp/20010128.freebsd-smp] was sent to the -smp mailing list.

12 January 2001

A status report [http://www.freebsd.org/cgi/getmsg.cgi?fetch=45111+48223+/usr/local/www/db/text/2001/freebsd-smp/20010114.freebsd-smp] was sent to the -smp mailing list.

11 October 2000

John Baldwin and Chuck Paterson came up with a preliminary list of rules [http://people.freebsd.org/~jasone/smp/smp_synch_rules.html] that should be followed when working on kernel synchronization.

8 September 2000

Here's a copy of an article [http://people.freebsd.org/~jasone/smp/smp_article] sent to BSDtoday and linked at Daemon News.

6 September 2000

The SMP code has been committed. All further work is being done in cvs rather than with patches.
An updated patch is available for download. This patch is proably what will actually get committed.

5 September 2000

An updated patch is available for download. This patch makes rtc a fast interrupt, uses locked instructions for mutexes in MP kernels, and corrects mtx_*() linkage within modules.

1 September 2000

The code is working for the most part now on i386 (UP and MP). Some additional coding is still necessary for the alpha, which is being done now.

30 August 2000

Updated patches for i386 and alpha are available here.

12 August 2000

Updated patches for i386 are available here. Process accounting still doesn't work correctly, but a number of other improvements have been made.

3 August 2000

Patches with functional heavy-weight threads for the i386 platform are available here. There are a couple of minor issues with this patch set. Specifically, process accounting doesn't work correctly.

6 July 2000

Sheldon Hearn has prepared a mutex(9) man page based on the BSD/OS one, which is available here.

5 July 2000

Jake Burkholder put an updated patch here.

3 July 2000

Jake Burkholder has the BSD/OS lock code working now, and has incorporated the pertinent portions of Matt Dillon's patches (idle processes, some of the schedlock changes, etc.). His patch set is available here.

26 June 2000

Chuck Paterson has provided the PostScript versions of his presentation slides for the first day and second day of the SMP meeting.

25 June 2000

Here's a copy of the SMP meeting summary that was posted to the -smp mailing list.

19 June 2000

Here's a copy of the SMP project announcement that was posted to the -current mailing list.

Page author/maintainer: Jason Evans <jasone@freebsd.org>