Today I spent some time reading about various test results that either involved a comparison of times for Windows XP vs Windows Vista, some that pitted a Mac vs a PC both using Vista, another which MS Office was quicker on this or that machine which made me wonder. Can any benchmark really be fair? Even when a tester really tries to be as fair as possible, are there other variables which can skewer the results no matter how careful the test is run?

Last week I mentioned a small test I made using Windows XP w/ SP3 and also Windows Vista w/SP1 installed as a dual booter on the same machine. I figured this would be a fairly representative test since the hardware was equal and so was the installed software for the most part. I concluded that XP was somewhat quicker to which I received a comment from one reader which stated on his machine Vista was quicker. How can this be? Is one of us fudging on the results? I don’t believe this is the case.

Then I realized something that I hadn’t given much thought about. What about the drivers? Since XP has been around for some 5 years, one could venture a guess that the drivers over this period of time have been fine tuned for the OS. Whereas for Vista, many of the hardware drivers are fairly new and may not be as finely tuned as we would hope for. On the flip side the reader who stated his Vista box was quicker than when running XP, may have the latest hardware with the latest Vista drivers and for XP the drivers may not be as fine tuned.

Anyone who had been involved with computers for any length of time realizes that it is different variables that makes one system faster than another even when both systems should be performing the same. Which comes back to the first question. Can any benchmark really be fair?

Ask AMD when they got caught altering some of their test results. I’m sure Intel has also done the same thing. So when it comes to operating systems, who can believe any of the test results?

After all. Does anyone really get 34 hwy 26 city that the sticker on the new car states? 🙂

Comments welcome.

[tags]benchmarks, trust, testing, results, skewered,  [/tags]