By Bruce Weiner
Mindcraft Inc.

June 29, 1999

The open benchmark conducted at PC Week Labs with the participation of Microsoft Corp. and representatives from the Linux community is not just another case of "benchmarketing." These tests provide valuable information about Windows NT Server 4.0 and Linux, including:

• The best-case performance of each operating system;
• The software tuning parameters used to achieve peak performance;
• The impact of various hardware configurations on overall performance; and
• The location of bottlenecks that limit overall performance.

There has been a great uproar in the Linux community over an earlier benchmark that Mindcraft published. The open benchmark audited by PC Week Labs' Pankaj Chowdhry and Henry Baltazar addressed the concerns the Linux community raised about testing procedures and tuning. Nevertheless, the results of the open benchmark support the same conclusion as Mindcraft's earlier test: Windows NT Server 4.0 is significantly faster than Linux.

Some in the Linux community, such as Red Hat, have taken these benchmarks as a wakeup call. They are finding performance bottlenecks in Linux and eliminating them. Others in the Linux community prefer to call into question Mindcraft; NetBench and WebBench, the Ziff-Davis Benchmark Operation's performance tests used as the basis for both the original and open benchmark; and benchmarks in general.

So, how can you, the IT professional, ensure you are always a benchmark winner? Simple. Know what's valuable to you. Following are three guidelines:

• Know what features of the system and software the benchmark tests and how your use relates to them. This will help determine if the benchmark is relevant for you.
• Know your minimum performance requirements, and consider that a benchmark will typically generate higher performance than you will experience in actual use.
• Study the hardware and software configuration tested, particularly the tuning parameters used. The tunes may be the most valuable information you get from a benchmark.

By looking at other benchmarks that test different aspects of a system and software, you can gather more information to help you make a purchasing decision. Benchmarks from industry consortia such as the Transaction Processing Performance Council and the Standard Performance Evaluation Corporation provide comprehensive tests that have been accepted by major industry participants.

Performance is only one dimension to your multidimensional purchase decision, albeit a gating one. For operating systems, there are several other important factors to consider, including training, development tools, application availability, system management tools and total cost of ownership.

Perhaps the most important thing you can take from benchmarks such as these is the time you save time learning about each product. Given the hyper-pace of business today, the time you save can be the difference between failure and success.

Bruce Weiner is president of Mindcraft Inc.

Copyright 1999