Tampering with benchmarks isn’t something new. A few manufacturers in the past have been caught doing it. It is nice to see a benchmark mathematically represent real world usage, but let’s face it, with the number of manufacturers optimising their devices for benchmarks, it is hard to derive real world performance from them. In the recent past when we at Digit get smartphones under embargo to review, benchmark access on these devices is blocked. We would like to believe that it is because these manufacturers don’t want their devices to be revealed through benchmark listings, but I guess that’s innocent wishful thinking. In a conversation with AnandTech Huawei official confirmed that it is using benchmark detection software to display the best possible results on the smartphone. The company says it is doing this to stay relevant to its competitors.
To quote AnandTech, “A single benchmark number, stated Huawei’s team, does not show the full experience. We also discussed the validity of the current set of benchmarks, and the need for standardized benchmarks. Dr. Wang expressed his preference for a standardized benchmark that is more like the user experience, and they want to be a part of any movement towards such a benchmark,” Dr Wang Chenglu is the President of Software at Huawei’s Consumer Business Group.
During the interaction, the Chinese smartphone maker has said that it has been working with industry partners for over a year. This is to find the best tests that closely represents a real-world user experience. “Dr. Wang also expresses that in relation to gaming benchmarking that ‘others do the same testing, get high scores, and Huawei cannot stay silent’.”
It is nice to see Huawei admitting to AnandTech that it is using benchmark detectors to stay relevant when compared to the competition, but there need to be some work done from the benchmark software makers to counter the tactics used by manufacturers. We have already seen the faking of camera samples by a lot of manufacturers, so it is hard to rely on the information given during a press briefing when a product is launched. Even if a manufacturer wants to tout some incredible capabilities of a smartphone, wolf has been cried too many times for us to take the information at face value.