Linux Scalability Project

Status report for January and February, 2000

Center for Information Technology Integration
School of Information at the University of Michigan

The primary goal of this research is to improve the scalability and robustness of the Linux operating system to support greater network server workloads more reliably. We are specifically interested in single-system scalability, performance, and reliability of network server infrastructure products running on Linux, such as LDAP directory servers, IMAP electronic mail servers, and web servers, among others.

Summary

Milestones

• IBM has agreed to fund work on thread performance in IBM's Java Virtual Machine for Linux. Their CAS graduate research program will sponsor a summer internship in Austin for one of our graduate students. Four graduate students in Brian Noble's 582 Operating Systems seminar are attacking the thread performance problem described in the paper entitled Java, Threads, and Scheduling in Linux. A project status report is available.

   

• Niels and Chuck are working on a paper about enhancements to poll() and kernel file descriptor management. The paper will also analyze the performance trade-offs between poll() and using POSIX RT signals to support asynchronous I/O. The paper will be published in the FREENIX track of the Summer 2000 Usenix Technical Conference.

   

• Because of the closeness of Linux 2.4, Chuck has decided to postpone his work on madvise() until the next development tree (Linux 2.5) opens.

   

• Steve continues to explore generic solutions to thundering herd issues in the Linux TCP stack. His accept() paper has been accepted for the FREENIX track of the Usenix Technical Conference for the summer of 2000.

   

• Chuck is reworking his malloc() performance report as a paper for the FREENIX track of the Usenix Technical Conference. The paper will include new microbenchmarks designed to assess the suitability of malloc() implementations for network servers.

   

• Andy Adamson and Dug Song continue their efforts on the Linux NFSv4 implementation. Several more VFS operations and parts of the Secure RPC layer have been completed. See the NFSv4 project site for project status.

   

• Claire Hough and Chuck are working on finding new ways for the Linux Scalability Project to connect with the evolving iPlanet organization.

   

• We are continuing to pursue research relationships with Red Hat Software, Silicon Graphics Incorporated, and the Stichting NLNet Foundation.

   

Challenges

NWFS is a Linux implementation of the Novell Netware file system done by contractors to Novell. During the project, these developers needed help integrating their work into Linux, and appealed to several well-known Linux kernel hackers for aid. It is not clear whether the kernel hackers ignored Novell or whether there were other reasons they did not respond. The result was mounting frustration on the part of the Novell contractors. Frustration gave way to a strongly worded series of messages directed to several kernel hackers, and copied to the linux-kernel mailing list.

While most dismissed the accusations as groundless drunken ramblings, many in the larger community remained silent on the issue. Mailing list members were probably offended by the harsh words and strident tone, as well as the audacity of the attacks on some of the most prominent members of the on-line Linux community, and thus ignored the deeper issues exposed by the messages. However, many have shared Jeff Merkey's experiences of little or no feedback from kernel developers, terse or even rude responses from some members of the community, and an almost arbitrary process for inclusion of new features in the mainstream kernel distribution.

Little feedback
Submitting patches and questions to a mailing list is risky. There is never a single recipient who is ultimately responsible for replying to questions or responding to patch submissions. Often, questions remain unanswered, and no feedback at all is provided to patch submitters, even after repeated attempts are made to submit a patch.

While not every submission deserves a response, professional developers do deserve an "in earnest" answer to why their submission is not good enough for the distribution. In most cases, provided such an answer, they can fix any problems and resend an entirely acceptible version of their patch. Questions deserve a response, even if it's "We don't know," or "That question is better addressed elsewhere." Granted, it's not the mission of the linux-kernel mailing list to answer all questions. However it is common courtesy to recognize submissions and to respond politely to questions. To do less than that is rude and off-putting, and Linux cannot afford the bad press. Whether or not old-hand Linux kernel developers like it, the linux-kernel mailing list is no longer a small group of people who know each other well.

Arbitrary and opaque process for inclusion of new features
What does "open software development" really mean? Is development open because everyone can participate, or because everyone can watch?

Rather than a democratic or consensus-based process, Linus Torvalds claims the Linux development process is really "a benevolent dictatorship." Linus owns the final decision about whether or not a new feature is included in the Linux kernel distribution. There are two main vehicles of communication amongst developers working on the Linux kernel. The first vehicle is the linux-kernel mailing list, and the second is the distributed source itself. The first is unmoderated, and the second is moderated by one man, Linus Torvalds.

Merkey has accused the Red Hat-employed developers of having some kind of "in" with Linus due to their financial backing. That's unlikely. What's more plausible is that Red Hat hired most of the experienced Linux kernel developers, those whom Linus trusts and is familiar with. Thus, he is more likely to take their code because he has already established a good relationship with them. He will even include new features from them after he has announced a code feature freeze.

What about the rest of us? There is no structured way to get a submitted patch into the Linux kernel distribution. Sometimes you send a patch to the maintainer of that section of code. Sometimes you post it on the linux-kernel mailing list. Sometimes you e-mail it to Linus. Sometimes you get one of the Red Hat developers to post it. Sometimes you write a paper or journal article about a problem, and print the solution with the article.

When all is said and done, there is no formal review process, no quality assurance, and no good way to get a patch onto the radar screen. The linux-kernel forum is noisy, chaotic, and inefficient precisely because of this. Can we really say that Linux is better off because of this process?

The communication path between the core kernel developers and independently working developers needs improvement. I've already posted some ideas for improving that process on this website. A long-term development plan, and especially a process for reaching consensus about it, would help the development community decide what features can be added in a single development cycle, and what qualifies each feature as ready for a stable release. Especially because Linux kernel development is a distributed process, it would be useful to have shared information about what is expected to appear in each major kernel release, who is working on which new features, and when the results of a project are expected to appear.

Having powerful debugging tools built in to the kernel would save kernel developers time and effort. It would provide the ability for the less skilled to make higher quality contributions to the code base, as well as to solve quickly some of their own difficult and frustrating bugs without requiring assistance from the already overworked gurus available in the Linux community. Some argue against the code bloat and the existence of debugging code in production kernels. In reality, kernel debuggers are small, and generally don't interfere with commonly used code paths. Keeping a debugging facility integrated at all times means that the facility can grow with significant changes to the kernel, rather than being an add-on that must be ported for each new kernel release.

Finally, making mechanisms for including modifications into the kernel distribution less ambiguous and less arbitrary would be a significant step towards lowering developer frustration. Everyone can play by the same "feature freeze" rules. The Maintainers list might be better served by a web-site that includes specific procedures required by each maintainer. A web-site might also allow the automation of patch submission so they automatically get to interested parties, and could simplify the process of creating a response to each submission. It could even add a more efficient and collaborative code review process.

Conclusion: whither a code of conduct?
In this community, new ideas are usually argued to death rather than openly and creatively considered, and their purveyors are treated with a great deal of sarcasm and disrespect. Some members of the Linux community become defensive at even the slightest thought that doesn't jibe with their constructed view of Linux. Many exhibit strong "Not Invented Here" tendencies. In other words, if it doesn't agree with The Vision, it is quickly and dourly dispensed with. Too often, suggestions meant in the most friendly and helpful of lights are dealt with severely and inappropriately.

After the dust from the NWFS e-mail had settled, a newer member of the mailing list suggested a code of conduct in response to Merkey's troubling tone. She was soundly trounced for the suggestion. As usual, most members of the list focused on the exact wording of her mail, and did not use any imagination in their interpretation of her words. Their response was swift, terrible, and not comfortable to watch.

It's never easy to take criticism, especially if you believe that all critics are out to destroy you. As Nietzsche says in Twilight of the Idols: "Was mich nicht umbringt macht mich stä rker" ( "Whatever does not destroy me makes me stronger." ) Many of its strongest critics are actually fascinated by Linux, and wish it well. But some of us can't see that.

It's time to truly open up the development process. It's time to tap the wide range of development resources and ideas that are becoming available to the Linux community. Unfortunately, there is no coherent and ultimate leadership in the community that can bring this about, since it is a community based on political and social chaos. Some even argue that is Linux's fundamental strength.

Copyright 2000