PC Week Labs' tests show what path Linux must take

PC Week Labs' tests show what path Linux must take

By Henry Baltazar and Pankaj Chowdhry
PC Week Labs

June 25, 1999

In a first-of-its-kind open benchmark comparing the performance of Linux and Windows NT, PC Week Labs not only found that NT remains substantially faster but also isolated Linux's shortcomings and gained insight into where future development of the open-source operating system should be headed.

These tests follow the controversial NT/Linux comparison conducted by independent benchmarking operation Mindcraft Inc. and sponsored by Microsoft Corp. The Linux community had cried foul after this April match, which found NT to be 400 percent faster than Linux. All the principals agreed to let PC Week Labs arbitrate a rematch. (See "The value of benchmarks" by Mindcraft President Bruce Weiner.)

After a tortuous five days of tests, audited by the best and the brightest from Mindcraft, Microsoft and Red Hat Software Inc., and despite significant tuning improvements made on the Linux side, Windows NT 4.0 still beat Linux using the Apache Web server and Samba in every performance category, although the margin of victory was smaller than in Mindcraft's tests.

But far more interesting is that, in all the areas in which the Linux community cried foul, its assumptions were wrong. Where kernel problems were found, fixes are already under way.

For instance, the open-source community objected to Mindcraft's use of the Apache Web server in its benchmarks, claiming that using Zeus, a Web server available commercially from Zeus Technology, would improve results. We tested Zeus on Linux and found its performance peaked almost exactly where Apache's did.

Working with Red Hat programmer Zach Brown, we traced the problem back to the lack of a multithreaded IP stack in the Linux networking subsystem, which caused a performance plateau in the operating system, not in the Web server.

The problem is being fixed in the next version of the Linux kernel, and a beta is available in the 2.3-kernel series. However, the upcoming improvements don't stop there. The next version of Apache will have a new static page engine, similar to the fast path in NT's IIS (Internet Information Server).

Zeroing in on the Web server

In the open benchmark using Ziff-Davis Benchmark Operation's WebBench 2.0, which Mindcraft used as well, the IIS 4.0 Web server pumped out an impressive 4,166 requests per second, compared with 1,842 requests per second on Red Hat's best run (see top benchmark chart).

In comparison with Mindcraft's initial report, IIS 4.0's performance was 126 percent better than Apache, as opposed to the 400 percent difference Mindcraft found. And where Linux's performance collapsed after the client load exceeded 16 computers in Mindcraft's original Apache benchmark, Linux fared better under heavier loads in our tests. The discrepancy in results may stem from the differences between the Mindcraft and PC Week Labs testbeds.

We also tested both operating systems on a single-processor box with 256MB of RAM to evaluate performance on lower-end hardware. NT still had a performance edge over Linux (1,863 requests per second compared with 1,314 requests per second in WebBench, for example). This amounted to a 41 percent performance difference but showed that, even on cheaper systems, NT came out ahead.

File server: Also not on top

Running ZDBop's NetBench 5.1 to test file service performance, we found that Linux running the Samba file gateway did not top NT in any test (see bottom benchmark chart). Linux and Samba's best numbers (which were achieved while running on NT workstation clients) were considerably slower than NT's best numbers (155.9M bps vs. 338.3M bps).

Previously, we had found that Samba could outperform NT 4.0 using NT workstation clients (see "NOS crossroads," PC Week's Shoot-Out of network operating systems). As a direct result of that Shoot-Out, Microsoft's performance engineers were prompted to find ways to enhance file throughput from NT servers to NT workstation clients.

Upon closer examination of the NTFS (NT File System), Microsoft found that when the data volume was configured as a single partition, the contention for NTFS' transaction log slowed file transactions significantly.

By breaking up the data volume into four NTFS partitions (each with its own transaction log), we were able to distribute transaction entries, eliminating the bottleneck.

Linux and Samba had their most favorable performance comparison with NT when we ran both on a single-processor server with 256MB of RAM. In this configuration, NT performed 52 percent faster than Linux running Samba (165.2M bps vs. 108.7M bps).

In ZDBOp's WebBench 2.0 test (top chart), which shows how fast a server can dish out static Web pages of varying size, Microsoft's IIS 4.0 Web server handily outperformed the Apache Web server running on Linux. When we switched to a uniprocessor server, the difference between Apache and IIS was much smaller.

In ZDBOp's NetBench 5.1 file serving test (bottom chart), the choice of clients did not make much difference, thanks to Microsoft's new NT tuning parameters.

Technical Analyst Henry Baltazar can be contacted at henry_baltazar@zd.com. Senior Analyst Pankaj Chowdhry can be contacted at pankaj_chowdhry@zd.com.