Preempting an expose, MediaTek wrote out a detailed blog post explaining and somewhat justifying why it does what it does to boost benchmark scores.
MediaTek does not believe they are cheating on benchmarks. Instead, they believe what they're doing is standard industry practice that’s also apparently practiced by its key competitors
Our tests prove that's not the case.
On Thursday, an AnandTech deep dive proved the existence of a ‘sports mode’ in MediaTek chipsets, using which smartphones powered by the chipsets post higher benchmark scores than normal. This was done by tweaking the code in the firmware to change the behaviour of the processor every time a benchmark app was detected.
As a result, every time you’d run a benchmark app, the scores certain MediaTek-powered smartphones would get would be unnaturally high, giving a sense of it being powerful enough to punch above its weight. If you can recall, this is similar to what Huawei had been doing previously. When tested with an anonymised version of benchmark apps, the numbers turn out to be far lesser than what it originally showed. In the case of MediaTek devices, the differences were as high as 75 percent in some workloads on PCMark Work 2.0.
Preempting the expose, MediaTek wrote out a detailed blog post explaining and somewhat justifying why it does what it does to boost benchmark scores.
Long story short, MediaTek does not believe they are cheating on benchmarks. Instead, they believe what they're doing is a standard industry practice that’s also apparently practised by its key competitors. Our tests show that that may not entirely be the case.
MediaTek’s side of the story
Apart from the statement, MediaTek also shared a few additional insights about benchmarking practices in the mobile industry. Broadly speaking, it outlines how every device is uniquely optimised to adjust for power and performance depending on the applications it runs. Here, MediaTek’s proprietary dynamic resource scaling comes into play to adjust the power consumed and performance delivered by the chipsets.
“This means that the device will only run full throttle (which eats up battery life) when it’s absolutely necessary for a good user experience. Our chipsets deliver a big power boost when the most demanding applications are being run so people can enjoy smoother gameplay and get the most out of the latest AI applications. On the other hand, our chipsets will operate in an ultra-power savings mode when you’re doing light tasks like checking email,” the company wrote in its blog post.
When is it absolutely necessary? Let’s say a highly resource-intensive game? How about PUBG Mobile? It’s known that the Tencent-made popular battle royale game is a resource hogger. You need a fast chipset to get a decent experience. We picked the MediaTek Helio G90T-powered Xiaomi Redmi Note 8 Pro and ran the AnTuTu test and a 4-minute session of PUBG Mobile, and clocked the performance using GameBench.
Here’s what we found –
Gamebench data showed the AnTuTu benchmark consumed an average 11.46% CPU. At its lowest, it was 8.83% with the AnTuTu app only taking up 0.32% CPU usage. Now check the CPU frequencies at the same time. All six efficiency cores are running at 1.8GHz while the two performance cores are clocking 1.9GHz, near about the peak speed of 2.05GHz that it advertises. And quite expectedly, when the CPU usage is its peak 100%, all eight CPU cores are clocking 1.9GHz.
If this flagship MediaTek chipset introduced with gaming in mind is running the CPU at its peak at all times during the benchmark run, MediaTek isn’t showcasing its prowess of dynamically allocating resources based on the workload. Instead, it’s simply asking the CPU to run at its highest possible speed, with nothing to keep the thermals from climbing up, to get a high score. In this case, the Redmi Note 8 Pro scored 285578, and during the time it launched, it was pegged as a game-changer for the mid-range segment for its gaming prowess. And looking at the scores, you’d have liked to believe it delivers one of the best gaming experiences?
Here’s how it performed during our PUBG Mobile run –
The four-minute session with all graphics settings maxed out, saw the CPU load at an average of 19.21% while at its peak, it was 58.87%. At its lowest, the CPU load given by the game was just 7.27%, and here the CPU frequencies were dynamically set to 1.3GHz for the efficiency cores while the performance cores were idling at just 795.6MHz.
On the other hand, when the CPU load was at its peak of 58.87%, the CPU frequencies had climbed up to 1.8GHz for the efficiency cores. The performance cores weren't even stressed running at just 1.2GHz.
The game itself ran at just 30FPS with an average 65% stability indicating that this isn’t the best performance, even with MediaTek’s dynamic scaling kicking in.
So if you bought the Redmi Note 8 Pro thinking the Helio G90T is a revolutionary chipset and for the mid-range segment and will offer stellar gaming performance, based on the benchmark scores, you would have been duped. Because the chipset that powers the smartphone isn’t quite capable of scaling up to the same level of performance that it achieves while benchmarking when you play PUBG Mobile. As a result, the frame rates are limited to what we usually see in the mid-range segment while the stability numbers are abysmally low.
Insinuating key competitors
In addition to justifying its practice of boosting benchmark scores, MediaTek also insinuated that it’s not alone in offering a ‘sports mode’. It’s key competitors also regularly indulge in such practices. We presume it’s talking about Qualcomm here.
Well, it’s true that smartphones powered by Qualcomm chipsets also offer a performance mode. The AnandTech article takes a Snapdragon 765-powered Oppo Reno3 that launched in China to show a ‘high-performance mode’ in the power settings. But here’s the thing. Unlike MediaTek, Qualcomm doesn’t keep this special performance mode hidden from users, triggered only when a benchmark app is started.
MediaTek has been hiding the code in plain sight and name-checking specific benchmarking apps (and only benchmarking apps) to trigger this so-called ‘sports mode’ that makes the memory controller and scheduler of the chipset far more aggressive than it usually is.
The company even went on to challenge AnandTech to review other devices powered by other chipsets to find out the same practice followed by them. We did the legwork for them. We took the latest Xiaomi Redmi Note 9 Pro powered by the Qualcomm Snapdragon 720G SoC and ran the same tests to see if we see the same behaviour happening here. Note that the Redmi Note 9 Pro does not have a ‘sports mode’ or a ‘high-performance mode’ or even a ‘monster mode’ that MediaTek claims exists in devices powered by other chipsets. Is it secretly working in the background?
Let’s check CPU states during the AnTuTu run –
The AnTuTu workloads follow the same pattern we saw during the previous run, and at its lowest the CPU load was just at 0.37%. Expectedly, the CPU cores were mostly idling with the efficiency cores running at 1.1-1.2GHz (two of them were turned off completely) while the performance cores were clocking a low 839.2 MHz. That’s in contrast to the G90T revved to its maximum.
And when the CPU load was at its highest, again at the same 6-minute mark, CPU usage hit a peak 100% and right on cue, all the CPU cores were fired up to their max frequencies. The Efficiency cores maxing out at 1.7GHz while the performance cores are clocking 2.2GHz.
Clearly, MediaTek’s claim doesn’t stand true in this case. Both chipsets advertise gaming as their forte, but where the Snapdragon 720G intelligently adjusts the core speeds based on the load on the CPU, the MediaTek Helio G90T simply keeps all taps open and lets the CPU run full throttle even when there’s nothing much for it to compute. The Snapdragon 720G during its run of AnTuTu scored 275632, which is far higher than what the G90T can manage.
The superiority extends in gaming as well. We played PUBG Mobile on the Redmi Note 9 Pro as well with all graphics settings maxed out, and the numbers looked far better than the Redmi Note 8 Pro. For one, it clocked in 30 FPS but at a higher 91% stability, indicating a far smoother performance with lesser frame drops.
The frequency distribution shows the Snapdragon 720G enters a high-performance mode when it detects a game. Not when it detects a benchmark app. At its lowest, the CPU was only taking up 0.37% load, while the cores still responded with high clock speeds. The chipset recognised I’m playing a game and effectively increased the performance of the chipset to ensure the game runs smooth. Similarly, at its peak usage of 46.18, the CPU had no thermal constraints kicking in, and continued running at peak frequencies taking in all the load the game was throwing at it.
Based on this, it seems MediaTek is partially right. Smartphones powered by its key competitors also have a secret ‘sports mode’, but in case of Qualcomm, the high tier kicks in when you actually need it; while gaming, which is when processors are the most stressed on a smartphone. The MediaTek Helio G90T triggers the sports mode when a benchmark app is detected, resulting in a high score. Meanwhile, the Snapdragon 720G's sports mode kicks in when you play a game, resulting in a better gaming experience. It's clear who has its priorities straight.
Benchmark apps are only used and enjoyed by a tiny subset of smartphone users. That includes us reviewers, product testers, app developers, and enthusiasts. On the other hand, MediaTek claims one in every three smartphones sold in the world is powered by a MediaTek SoC. That’s a large number of smartphone users and most of them won’t even download a benchmark app to see how fast their shiny new smartphone runs. They will play a game or multitask with social media apps, that’s where they will need the extra performance.
More than anything, what this lengthy blog post insinuating the work of an acclaimed publication really indicates is that MediaTek will move a lot of boulders to address the bad press, but won’t really do anything to improve user experience.
Follow Us
Subhrojit Mallick
Eats smartphones for breakfast. View Full Profile
