AMD Calls Out Intel, BAPco For Benchmark Cheating
Once upon a time, benchmarks have been the favorite "authority" when choosing products. But thanks to a scandal that rocked the mobile world a few years ago and, more recently, the automobile industry, the objectivity and infallibility of such tests are now being put into question. Taking advantage of recent hurts, AMD is once again pointing the finger Intel, as well as the BAPco benchmarking consortium, for unrealistic, misleading, and almost mythical benchmark results that unsuprisingly put Intel ahead of AMD in terms of CPU performance.
In a video that almost fits a crime documentary, AMD puts an Intel Core i5 computer to a few tests against a "comparable" yet unnamed AMD PC on the FX platform. First, using the BAPco's SYSmark tool, the Intel machine garnered a score of 987 while AMD only got 659, a delta of 50 percent that AMD just couldn't stomach. So they turn to FutureMark's PCMark software, where Intel scores 4199 and AMD got 3908, a delta of 7 percent only. Third time's a charm, they say, so AMD cooked up a script that timed the two PCs took to complete a task. The Intel PC finished in 61 seconds. AMD's took 64 seconds.
In short, something fishy is going on with SYSmark, AMD is effectively saying. It is not a reliable objective standard, which is why the FTC in 2010 required that SYSmark have a disclaimer that it was optimized for Intel processors.
Amusingly, AMD doesn't contest the results that show that, in all cases, it comes up short against Intel, no matter how big or small the delta. In fact, it even hints at why SYSmark's Intel scores are way off the chart. SYSmark apparently benchmarks and taxes the CPU particularly while something like PCMark also takes into account GPU and other components. Perhaps Intel is really only interested in the CPU, which is its primary product anyway.
AMD is effectively opening up old wounds with this new video. In the early 2000s, AMD joined the BAPco consortium, a non-profit group that aims to develop and distributed objective performance benchmarks. The consortium includes HP, Intel, Lenovo, Microsoft, Samsung, and, at one point, even Apple, just to name a few. In 2011, AMD made a lot of noise as it left BAPco, accusing the consortium of swaying to Intel's side.
Benchmark tools are really a tricky subject these days. Even those third party tools that promote objective can be cheated to some extent. Not to mention differences in opinions of what benchmarks should really test and how. They're not totally useless, as they can give an idea of both theoretical and actual performance of systems, but they should never be taken as absolute and irrefutable proof.
VIA: Ars Technica