Microsoft’s Muddled OS Test
Questions Arise Over Methods in Tests of Windows NT and Linux
By Michael J. Martinez
ABCNEWS.com
April 28, 1999 — You take two different pieces of software and run them on the exact same computers, then you compare the results. Simple, right?
This process is called benchmark testing, and it’s one of the few ways to truly compare competing software products.
Last month, Microsoft decided to do a benchmark test of its Windows NT operating system vs. its newest, boldest competitor, the open-source Linux operating system. Microsoft contracted with Mindcraft, a Los Gatos, Calif., testing firm, to run the comparison, and when the results were announced, Windows NT was the clear winner.
Normally, that would be the end of it. But the community of hackers and developers that built Linux was furious. While Microsoft engineers spent a great deal of time “tuning” the Windows NT test computer — making sure it would run the test software at its highest efficiency level — Mindcraft installed the Red Hat Linux Operating System practically out of the box, with no tuning whatsoever.
Microsoft paid for the test, hosted the testing machines in its own labs, tuned and tweaked its machine — and won.
Was this a valid test, skeptics wonder, or an attempt to spread fear, uncertainty and doubt (FUD, in tech parlance) about Linux
Top Secret Tech Support?
Mindcraft’s president, Bruce Weiner, now knows the depth of passion that the open source community has for its favorite software — and his mail server knows it, too. Ever since releasing the test results, Weiner has been flooded with flame mail, accusing him of pandering to Microsoft and failing to seek out advice on how to tune a Linux computer.
“Yes, Linux could’ve been tuned better, and the results would have been somewhat better,” Weiner says. “But this information is not readily available. Someone really needs to document the best-kept secrets on tuning Linux systems. It would be a handy thing to have.”
Although Mindcraft turned to Linux distributor Red Hat Software for help before testing, Weiner says he received no response to his inquiries. He also had a colleague post questions on Linux newsgroups, but the responses weren’t workable. Respondents, he says, urged him to change his hardware; but since the hardware was key to the testing, this was impossible.
“It was only after we published our first test results,” says Weiner, that Mindcraft got the help it needed. Now the company is doing a second head-to-head test and plans to release the results in a couple of weeks. Weiner expects the results to be similar to the first test.
Will Second Time Be the Charm?
For this second test, Weiner has assigned what he calls “unimpeachable Linux experts” to prepare the Linux computer. Though he wouldn’t identify the experts, both Red Hat Software and Linux creator Linus Torvalds say they’ve been in touch with Mindcraft.
But Torvalds has his doubts that the second go-round will go any better. “I still don’t think that another test will make this right,” he says. “We helped them out, gave them a few more things to tune, but they wouldn’t let us in the lab, and they wouldn’t answer our follow-up questions. It became so opaque for us, there wasn’t much we felt we could do.”
Torvalds notes that previous comparisons run by Mindcraft for Microsoft showed similar results against other operating systems, such as Sun’s Solaris and Netware.
“I think if you take Microsoft engineers and tune NT just right, it could beat anything you put against it,” Torvalds says. “In many of the other, independent tests we’ve seen, Linux just beat the hell out of NT.”
Testing by PC Week last month seems to back him up. When the magazine compared file serving software, Linux kept up with NT, while Linux easily beat NT in Web serving. Weiner, in response, questioned PC Week’s testing methods, noting that they weren’t published.
Other questions have arisen about the validity of Mindcraft’s benchmarking run. For one, e-mail from Weiner and other Mindcraft testers originates at a numerical IP address that belongs not to Mindcraft, but to Microsoft. That left Torvalds wondering just how impartial Mindcraft could be.
Weiner did not return calls seeking comment on his company’s relationship with Microsoft.
The Thrill of Victory
Microsoft product manager Ed Muth says the company was pleased with Mindcraft’s report, and is satisfied that the testing firm did everything it was supposed to do.
“We’re thrilled to see this good performance, but not surprised,” Muth says. “We could, of course, say everything was fair and aboveboard, and we think it was. But I think we’ll wait until the new benchmark tests come out. If the results are unchanged, that will be a more powerful statement than anything else.”
Muth adds that the Mindcraft test was spurred by previously published benchmarks, which did indeed show that Linux kept up, and in some cases surpassed, Windows NT. The Mindcraft test, however, was done with a high-end, multiple-processor machine, which NT was tuned for. Even Torvalds admits that Linux might not be ready for such massive computing power.
“I use a multiprocessor machine at home, and it handles my workloads fine,” Torvalds says. “But on really big machines, this is an area where Linux is just entering now.”
So what did the Mindcraft test achieve? It gave Microsoft strong numbers to point to — but at the cost of a public-relations embarrassment.
Copyright ©1999 ABC News Internet Ventures.