Opinion: A tale of two scores

Every now and then, when we’re running benchmarks on finished goods such as laptops, phones and even individual components such as graphics cards, we’d come across a score that simply does not match expectations, rather it leaps over the expected numbers. Not all manufacturers indulge in such blatant fraud. Some are a bit subtle and don’t inflate their numbers that much. Competition makes people cross ethical lines. We’ve seen so many brands try to pass off DSLR images as photos taken by the phone camera. Even more brands utilising hidden power profiles that only kick in when a benchmarking application is running. It’s a very human thing as we’ve seen recently with the professional CS: GO scene in India. Upon getting caught, some deny any wrongdoing, some confess and some act stupid and try to cover it up with asinine actions or reasons. 

For big brands, it’s all about ensuring your product sells more than the competition. Since the average consumer is none the wiser, brands get away with these tricks. Even if they were caught, they aren’t ashamed. Some brands have a very simple reason to give – others are doing it as well. In a way, we can see the logic that they’ve used. Since the average consumer reads a few reviews before making his/her purchase decision, if mobile A gets a higher score than mobile B, then few consumers will go for model B. In the consumers’ mind, model B is providing more value for their hard earned money. If brands don’t boost their scores artificially, then they will lose out on sales and subsequently, the bottom line suffers. For some brands, this could very well affect their survival in the extremely competitive Indian market. However, most of the cases of benchmark cheating are done by established brands. This speaks of greed on part of the established brands. 

With more reviewers becoming aware of the benchmark cheating issue and the emergence of new benchmarks that are not as popular as the big three, we’ve begun to see more brands getting caught of late. Soon enough, cheating in benchmarks will become enormously difficult and the playing field will be levelled once more. At this point, we’re pretty certain that brands will try to pass off these performance boosts as anomalies brought on by some “AI” feature or the other. 

When a brand cheats in such a manner, they are the first to lose consumer trust. The manufactures who did cheat also affect the those who are not cheating since they appear to be way behind the competition. So unless Government bodies step in and slap on anti-trust lawsuits on these brands, there’s little reason for them to stop. As of now, it appears as if every brand cheats in some manner or the other and not cheating is the “odd behaviour”. Soon enough, if nothing’s done, this kind of behaviour will be normalised and getting away with misinformation will cease becoming an offence. We shudder to think what a world that will be where the brand that wins does so on the basis how well they can cheat rather than being decided by how well they perform. 

As of now, benchmarks on phones are no longer trustworthy. Should you need a performance reference point then you can simply take a look at the SoC benchmarks released by the SoC manufacturers. Another course of action is for benchmark developers to make it more difficult to detect their presence on a phone or PC. One way to achieve this is to randomise the signature characteristics of the app. As of now, it’s the easily detected information such as package name that helps phones switch to a different profile. If such information can be obfuscated, then we might see the rampant cheating come to an end and thus, see true innovation arise. One can only hope. 

Follow Us

Agent 001

I have a keyboard and I'm not afraid to use it, because I have a license to quill. View Full Profile