FutureMark press release, May 23rd
(http://www.racesimcentral.net/)
"After the launch of 3DMark03 Build 320 Futuremark has received reports from
the members of its BETA Program concerning certain anomalies with 3DMark03
and Nvidia drivers. ExtremeTech (www.extremetech.com) has published an
article on suspecting NVIDIA drivers to improperly boost scores on
Futuremark's 3DMark?03. Some of these anomalies have also been reported by
Beyond3D. Alarmed by all these reports Futuremark has conducted a thorough
internal audit regarding this matter and has verified that certain NVIDIA
drivers indeed seem to have detection mechanisms, which are triggered by
components of the 3DMark03 program. We have identified eight such
mechanisms.
"In our testing, all identified detection mechanisms stopped working when we
altered the benchmark code just trivially and without changing any of the
actual benchmark workload. With this altered benchmark, NVIDIA's certain
products had a performance drop of as much as 24.1% while competition's
products performance drop stayed within the margin of error of 3%. To our
knowledge, all drivers with these detection mechanisms were published only
after the launch of 3DMark03. According to industry's terminology, this type
of driver design is defined as 'driver cheats'."
Sounds like a pretty harsh allegation. Here's nVidia's response:
"Since nVidia is not part of the Futuremark beta program (a program which
costs of hundreds of thousands of dollars to participate in), we do not get
a chance to work with Futuremark on writing the shaders like we would with a
real applications developer. We don't know what they did, but it looks like
they have intentionally tried to create a scenario that makes our products
look bad."
Okay, so FutureMark receive reports from BETA Program members about nVidia
cheating, but nVidia is not a member of the BETA Program, so who is telling
the truth? Lets see what ATI had to say about this, being the main
competitor to nVidia and also a member of the BETA Program:
"Despite still being a full Futuremark Beta member, ATI did not make it out
of the report entirely unscathed either. There is a performance difference
of about 8 per cent in Game Test 4, that accounts for about a 2 per cent
difference in the final 3DMark03 score, between the new and old versions,
indicating that although not visually different something was occurring on
this particular test.
"The 1.9 per cent performance gain comes from optimization of the two DX9
shaders (water and sky) in Game Test 4 . We render the scene exactly as
intended by Futuremark, in full-precision floating point. Our shaders are
mathematically and functionally identical to Futuremark's and there are no
visual artifacts; we simply shuffle instructions to take advantage of our
architecture. These are exactly the sort of optimizations that work in games
to improve frame rates without reducing image quality and as such, are a
realistic approach to a benchmark intended to measure in-game performance.
However, we recognize that these can be used by some people to call into
question the legitimacy of benchmark results, and so we are removing them
from our driver as soon as is physically possible. We expect them to be gone
by the next release of CATALYST."
Hmm, it seems ATI were doing the same thing, but because they are a member
of the BETA Program, nobody said anything. Fair play to ATI for not just
playing off of potential nVidia embarrasment.
So, anybody know why FutureMark are so upset with nVidia?
Nick