Open Benchmark Invitation

Rationale and Invitation

Linus Torvalds has been quoted in a few stories (ABCnews.com ,Linux Today, and Salon) as expressing concern about a new benchmark that Mindcraft was running. Mindcraft's honesty and name have been impugned in these stories and others as well as in great deal of newsgroup postings and email we have received. To get the other side of the story, please look at our rebuttals to the ABCnews.com, Linux Today, and Salon articles.

Mindcraft believes that the best way to continue to show our name as a credible source of information is to have an Open Benchmark. Therefore, we welcome the opportunity to perform a benchmark of Linux and Windows NT Server that is open to the best experts in the Linux community. Mindcraft will participate in this benchmark at its own expense.

Mindcraft has withheld the publication of our second Linux and Windows NT Server benchmark results (the test for which Linus and others provided some suggestions for tuning) pending the response to this Open Benchmark invitation.

We will be glad to conduct this Open Benchmark at any mutually agreeable test site. As one option, Microsoft has volunteered their test lab where Mindcraft conducted its previous tests.

We call on Linus Torvalds to invite anyone he chooses to tune Linux, Samba, and Apache. We also invite Red Hat to send anyone they choose to participate in the benchmarking as a Linux Expert. In addition, we invite Microsoft to tune Windows NT Server. The Linux Experts, Microsoft, and Mindcraft will witness all tests.

Purposes

1. To see if Mindcraft's second benchmark results are biased and not representative of Linux's true performance.
2. To do a fair comparison of Linux and Windows NT Server 4.0 with Linux tuned by Linux Experts and with all testing witnessed by them.

Test Environment

The following test environment will duplicate as closely as possible the one Mindcraft used for its second test of Linux and Windows NT:

1. Mindcraft will arrange to have available the same system that was used for the second Linux/Windows NT Server test.
2. The tests will use 4 x 100Base-TX networks for all configurations.
3. The tests will use Windows 9x clients (these can be in any mix including all clients using the same Windows 9x OS). The same client set up will be used for both Linux and Windows NT Server.
4. Samba and NT file sharing tests will be done using the NetBench Enterprise mix. Only the number of clients may be changed from the standard Enterprise mix so that a sufficient load can be generated on the server. The same client set up will be used for both Linux/Samba and Windows NT.
5. Apache and IIS Web server tests will be done using the WebBench zd_static_v20.tst, modified only to account for the number of clients used. The same client set up used for NetBench testing will be used for both Linux and Windows NT WebBench testing.

Test Procedure

For the tests labeled "Mindcraft Config" in the Tests table below, neither OS can have tunes, patches or bug fixes applied that were not used in or that were not available at the time of Mindcraft's second test. For the tests labeled "Linux Experts Config," the Linux Experts may apply any tunes they want, including patches or bug fixes that were available at either the kernel.org or Red Hat Web sites on April 20, 1999, the date Mindcraft started its second test.

The Linux Experts can run any of the tests in the Optional Tests table below that they want using the same client setup as the other tests.

1. The Linux Experts will load Linux 2.2.6, Apache 1.3.6, and Samba 2.0.3 on the server. They will use Mindcraft's configuration for each and will set the tuning as they gave it to Mindcraft.
2. They will run the NetBench and WebBench Linux tests named "Mindcraft Config" in the Tests table below.
3. The Linux Experts will tune Linux, Apache, and Samba as they choose and run the NetBench and WebBench tests named "Linux Experts Config."
4. If the Linux Experts want to do the Optional Tests, they can use any Linux kernel version, Apache version, Samba version, patches, and bug fixes posted on the Web prior to the start of the Open Benchmark.
5. Mindcraft or Microsoft will load Windows NT Server 4.0, Service Pack 4, and IIS 4 and configure them as in the second test.
6. Mindcraft will run the NetBench and WebBench Windows NT tests named "Mindcraft Config" in the Tests table below.

Tests

NB=NetBench WB=WebBench

Optional Tests

Publishing the Outcome

Mindcraft, any of the participating Linux Experts, and Microsoft will receive the raw test results and will have unrestricted use of the test results.

1. Mindcraft and any of the Linux Experts that want to participate will issue a joint press release describing the test results. The tone of the press release will include the test results, will be factual in tone, and will be positive about the opportunity to have an Open Benchmark with the Linux experts involved. There will be quotations from Linus Torvalds or his designee, Red Hat (if they want to participate), and Bruce Weiner (Mindcraft).
2. Mindcraft will issue a report on its Web site similar in structure to the one for its first report.
3. Red Hat and any of the Linux experts may generate their own reports regarding the test.

Please address any questions or comments to Bruce Weiner, bruce@mindcraft.com .

Copyright 1999