Fallout From Mindcraft Survey Leads to Rematch, Bake-off

Fallout From Mindcraft Survey Leads to Rematch, Bake-off

By Rachel Chalmers
Computergram International

May 7, 1999

Controversy over a Mindcraft Inc survey, which claimed to prove that NT is five times faster than Linux as a file server (CI No 3,647), just will not lie down and die. The open source community was quick to ridicule the first test for bias and ineptitude. Linus Torvalds himself said: "The number one reason [Linux lost] was really that the benchmark was selected for NT, and depending on how you select the benchmarks, you can always get the results you need... As it is, we don't much respect the integrity of the Mindcraft guys."

At the end of April, Mindcraft responded to its critics by announcing a second test, this time relying on the Linux community's "top minds". Those top minds were less than impressed with the whole exercise. In an article for LinuxToday, key Linux developer Alan Cox wrote: "I have basically given up on Mindcraft's Microsoft funded pranks. The careful use of the word 'support' to imply we somehow validate his test is misleading. I sent him about four emails... End users have a right to honest, open benchmarks they can trust when selecting a product. It is sad that the prospect of an open choice seems to scare some large vendors."

Mindcraft has not yet released the results of the second test, claiming to be working instead on a third, truly-open benchmark. This time, to squash those pesky accusations of bias once and for all, representatives of Linux will be encouraged to be physically present at a neutral site. But Torvalds and Cox are reluctant to participate. As Linux Weekly News points out, even Mindcraft's suggestion for a neutral site is suspect. It's the lab, owned by Microsoft Corp, where the original benchmarks were conducted.

Meanwhile Bruce Perens, another prominent figure in the open source and free software communities, has proposed a bake-off at LinuxWorld in August. "We will challenge NT and Linux developers to each field teams to compete at LinuxWorld," Perens wrote to the Linux International discussion list. "The two teams get identical hardware provided by a non-competing third party: Dell, IBM, whoever."

Each team, he suggests, should have a day or two to bootstrap and tune their systems. Then they would compete on speed and on a variety of rapid application development tasks, including bugginess, features and elegance. Perens wants the action to take place on the show floor, and be covered in live webcasts. "It's sort of like Deep Blue vs Kasparov," he concluded. "You win on the publicity front even if you lose." That's more than Mindcraft can claim.