Computer Database Plus                                    Article Selection Menu




   1  Benchmark cheating tarnishes the industry's reputation. (graphics board
      vendors; Winbench testing) (Tech Talk) (Column), InfoWorld, Feb 22, 1993
      v15 n8 p37(1).
      Reference # A13433952   Text: Yes (3314 chars)   Abstract: Yes

   2  New GUI accelerators featured at Comdex; Trident storms already crowded
      market - Matrox to follow. (graphical user interface) (Comdex-Fall 1992,
      Trident Microsystems Inc.'s Storm accelerator, Matrox Ltd.'s MGA two-chip
      accelerator) (includes related article on prices and availability)
      (Product Announcement), Microprocessor Report, Dec 9, 1992 v6 n16 p13(5).
      Reference # A13042745   Text: Yes (22555 chars)   Abstract: No

   3  PC MagNet. (shareware for Microsoft Windows 3.1 graphical environment)
      (Product Announcement), PC Magazine, Nov 10, 1992 v11 n19 p497(1).
      Reference # A12776765   Text: Yes (6891 chars)   Abstract: Yes



Enter as many as  3 choices (? for help) ! 1
Citation:    InfoWorld, Feb 22, 1993 v15 n8 p37(1)

--------------------------------------------------------------------------------



Title:       Benchmark cheating tarnishes the industry's reputation. (graphics
             board vendors; Winbench testing) (Tech Talk) (Column)

Authors:     Gibson, Steve

--------------------------------------------------------------------------------



Subjects:    Benchmarks;  Graphics boards/cards;  Manufacturers;  Performance
             Analysis Software;  Testing;  Ethics

Products:    WINBENCH (Performance analysis software)_Usage

Reference #: A13433952

================================================================================




Press <CR> for more (? for help) ! s

Citation:    InfoWorld, Feb 22, 1993 v15 n8 p37(1)

--------------------------------------------------------------------------------



Title:       Benchmark cheating tarnishes the industry's reputation. (graphics
             board vendors; Winbench testing) (Tech Talk) (Column)

Authors:     Gibson, Steve

--------------------------------------------------------------------------------



Subjects:    Benchmarks;  Graphics boards/cards;  Manufacturers;  Performance
             Analysis Software;  Testing;  Ethics

Products:    WINBENCH (Performance analysis software)_Usage

Reference #: A13433952

================================================================================



Abstract: Widespread manufacturer cheating of the Winbench testing standard for
          measuring Microsoft Windows graphics performance hurts the graphics
          board industry's credibility. Several companies have been caught
          cheating by performance analysis laboratory personnel. Many companies
          believe that widespread cheating by competitors forces them to cheat
          as a means of remaining competitive. Employees of companies that cheat


          on performance tests fear that they will lose their jobs if they
          disclose the cheating. After Winbench was upgraded to version 3.1, the


          scores of many graphics boards fell significantly only to improve up
          to 300 percent within just a few days, indicating that these companies


          had found a way to beat the tests. It is the job of the computer press


          to watch for companies that cheat on benchmark tests and it is fair
          for the press to reveal the names of companies whose cheating can be
          proven.

================================================================================



Full Text COPYRIGHT InfoWorld Publishing Company 1993

Last week, when I shared a recent true story of Windows display benchmark
cheating, you probably noticed that I avoided disclosing the cheater's identity.


Much as I abhor the idea that this well-known and significant company would
deliberately manipulate its Windows display drivers for the express purpose of
fooling its customers, I believe it would not have been fair to single this
company out from the crowd, given that I've also found and confirmed four other
cases of recent Winbench cheating.

In fact, the smaller company in last week's story, which blew the whistle on the


larger cheater, was itself guilty of Winbench cheating not very long before. The


company has since cleaned up its act, and it now has very good drivers that
appear to be truly fast.

As I've schmoozed around the Windows display accelerator industry, I've picked
up many tidbits of information. I've heard that everyone has, at one time or
another, been "forced" into cheating on the Winbench. And I've listened to
desperate marketing people when I've caught them red-handed -- with their
drivers down. They moan that they just didn't have any choice but to cheat.

If your competitor is cheating on the benchmark in order to attain a great
reputation with a super-fast display adapter, huge market share, and OEM
contracts, what do you do? Cry wolf and point your finger? Then everyone assumes


that it's just sour grapes because you happen to have a "much slower" display
adapter, when in fact you just chose the painfully honest path. How many people
are so committed to honesty that they'd be willing to accept unemployment for
themselves and their entire company rather than tweak their drivers to yield a
better score on a dumb freeware benchmark?

One of the problems, of course, is that the Winbench is both a very popular and
a very "cheatable" benchmark. Neither Version 2.5 nor Version 3.1, released last


fall, use the system's display drivers the same way Windows does. So it's very
easy to alter a poorly performing Windows driver to earn a high Winbench score.

I heard a story of a little-known Windows display adapter company that showed
one of the industry's testing labs an "unbelievably amazing" display adapter,
which scored several tens of millions on the Winbench. The lab took one look at
the card, recognized its antique chip-set technology, and knew there was no way
those chips could do what was being claimed. The lab politely excused the
company's representatives.

But how many users would recognize the numbering on the chips and realize that
the only thing the "miracle" card they'd just bought was good for was running
the Winbench?

When the cheater-beating Version 3.1 of Winbench appeared on the first day of
Comdex, most display adapters' scores dropped to the floor. They'd been caught
by the Winbench changes. But several adapter makers, aware of the marketing
importance of a high Winbench score, put their engineers to work immediately. By


the end of the show, just several days later, a handful of the scores showed a
300 percent improvement under the new Winbench! They had "recovered"
substantially from the changes to Version 3.1, and their cheating code was once
again alive and functioning under 3.1.

While it is not my job to be the industry's policeman, you depend upon my
research and explorations to guide your purchases. Moreover, I'm uncomfortable
with not sharing the information I've uncovered about benchmark cheating in our
industry. Therefore, this column places the entire display adapter industry on
notice: Any company whose current display drivers I find to be cheating after
May 1 will be disclosed right here in print. You have more than 60 days to clean


up your acts, after which time I will hold you accountable. In the meantime,
buyers, beware high Winmarks!

Steve Gibson is the developer and publisher of SpinRite and president of Gibson
Research Corp., based in Irvine, Calif. Send comments to InfoWorld at MCI Mail
259-4127 or fax them to (415) 358-1269.

================================================================================



Last page (? for help) !
Computer Database Plus                                    Article Selection Menu
