The Nexus 4 here at the lab has been flashed and is Lollipop ready. So, we thought we would carry out some comparisons with another Nexus 4 running on KitKat.
You’re probably familiar with the UI changes that Google brought to Android in its Lollipop update. Android 5.0, though, is not called the ‘biggest’ update to Android for those changes alone. In fact, perhaps the biggest change that Google has made here is in switching to the ART runtime instead of Dalvik, which was the default choice till KitKat. Google had actually been inching towards this since KitKat, allowing users to change to ART from the developer options in KitKat.
We used the Nexus 4, which was launched originally on Jelly Bean and got updates to KitKat and now Lollipop. What we aim to find is by how much, or if at all, the performance has increased.
What’s the difference?
Dalvik processes code ‘just-in-time’, while ART does so ‘ahead-of-time’. Now that we have the jargon out of the way, here’s what it really means. ART will improve performance of your apps in areas like time taken to launch applications, load times etc. In addition, ART also brings better garbage collection, which is better for apps that are prone to frame rate drops.
What’s the impact supposed to be?
As mentioned above, ART is supposed to improve the performance of your phone, allowing for faster processing and better garbage collection. In practice, you probably won’t feel any difference while playing games, but app launch times should be considerably reduced. What we set out to find was whether this was true.
Statutory Warning
The tests have been performed on the Nexus 4, which has pretty old hardware. The results may very well vary on newer devices like the Nexus 5 and others, although Lollipop would still come out on top. What would differ though is the margin by which Lollipop beats KitKat.
Benchmarks
We used everything from multicore tests to browser benchmarks, with Lollipop coming out on top every time. We ran a total of six different benchmarks, each running multiple times in order to ensure that there were no anamolies. The devices were also rebooted before the tests, so as to give it a fresh start for each test.
According to the benchmarks, the Nexus 4, which was originally launched on Jelly Bean shows a 48% performance increase for regular activities with Lollipop. The improvement is much lower when compared with KitKat, but there’s still about a 6.5% increase. On graphics tests, the improvement is about 2.5% from KitKat to Lollipop.
Regular Usage
This is where it gets trickier. The ahead-of-time code processing on ART means that your apps should take more time to install, but less time to load. To the naked eye, this is barely visible, but Lollipop is often faster. We had to provide for things like data speeds and heat levels, which led us to the fact that ART is indeed faster. Putting the two Nexus 4 devices side by side, it’s quite hard to figure out the difference, but keep at it and you will see that the Lollipop running device is often faster.
You have to realise that the difference in load times is in milliseconds, which means that the human eye can often not perceive the difference. In the long run though, a Lollipop device will allow for better multitasking. The memory management is better on Lollipop and so is garbage collection, which allows for faster and hence better performance. That said, we saw much longer boot times for the Lollipop powered Nexus 4 as compared to KitKat. In some places, apps were also faster on KitKat.
In newer devices that take advantage of ART and the 64-bit support, the difference will also show up when playing games, as ART allows your games to run longer without dropping frames. Why? Firstly because of better garbage collection. In the Dalvik runtime, there used to be two garbage collection pauses, while ART reduces that to one, performing a full cycle only when the device is locked. In terms of benchmarks, the Nenamark benchmark showed 56.8 fps to 58.8 fps in KitKat and Lollipop respectively.
Lastly, we also saw that the KitKat powered Nexus 4 heated up faster and to a higher degree as compared to Lollipop. CPU temperature testing apps show that while the Lollipop powered phone remained within 45-50 degrees after rigorous benchmarking, the KitKat device went up to 60 degrees. While we are attributing that to Lollipop as well, it could very well be a device issue.
Overall, there's not a huge difference in the performance of the Nexus 4 and you might even feel that it has gone down a bit. The reason for this is most likely the older hardware and not the software itself. One of the first changes we noticed is that the Lollipop powered smartphone took longer to boot as compared to KitKat. In addition, apps will take more time to install, since the compilation is done at install time in order to improve the performance of the app later. While the Nexus 4 did perform better on some apps, the difference wasn't enough to talk about.
Battery
Battery life has always been a big question mark in the smartphone industry. While we have come a long way from the first Android or iOS devices, there’s always room for more. Lollipop was also supposed to do more.
We ran our usual battery test on both the devices and while the KitKat powered device gave us 5 hours on the video test, the Lollipop running Nexus 4 went up to 5.5 hours. We weren't able to reproduce the two hour battery increase that we had seen on the Nexus 7 (2013) when we had tested the battery on it with the Lollipop developer preview. The extra 0.5 hours that we're getting is probably a result of Adaptive Display as the battery test shows the same results with the feature turned off.
This isn't the big update that Project Volta had promised us, but it's still an upgrade. Unlike many other things in Lollipop though, Volta requires a lot of developer support. Developers need to be responsible and make apps a certain way. There is a new API, which tells applications to update only when your phone is both connected to a WiFi network and charging. If you have been using Google's Photos app earlier, you would have noticed the same thing happening with your photos backup on the cloud. In Android, your apps keep waking up the device for various purposes, even when you're not using your phone. The new API (Jobscheduler) stops this and allows developers to code apps in a way that only allows for things to happen when certain criteria are met, instead of checking and failing again and again.
Notice that AnTuTu is waiting for an update in the screenshot below, even though the phone is connected to WiFi.
There is also the much touted battery saver mode, which brings down the processor speed and restricts data in order to provide better battery life. There's not much to be said about this, except that it works pretty much the same way as most others solutions out there. You will have to individually open apps in order to get notifications and your phone won't be sending you mail, WhatsApp and other such notifications while in the battery saver mode.
Bottomline
That said, our tests show that Lollipop is better than KitKat, but how different manufacturers leverage this difference is what will decide its usefulness. Just like Apple's iOS 8 platform, Lollipop is also offering a lot to the developers and the ART runtime, though better, won't be seen in its full glory by users till developers and OEMs start using it. Many of you might question why the comparison is being drawn to iOS, but one must realise that Lollipop is still making a two year old flagship better in some areas, while iOS 8 brought a lot of older Apple devices to a standstill. The difference is not much right now, but there's a lot that can happen in the future. In addition, it's going to be interesting to see how much the improvement is in the lower end devices like the Moto E and Android One smartphones when Lollipop comes to them.
It won't be surprising if Google rolls out a 5.0.1 update to the OS in a few weeks. Moreover, developers will start playing around with the OS, using things like better memory card support and clearing up the various areas that we are still in the dark about. On the face of it, Lollipop looks like an incremental update, but there's a bigger picture here, one that will reveal itself with time. For example, imagine the Moto G becoming the benchmark for sub-10k phones. There's a lot to look forward to and that is why this is the 'biggest' update to Android till now.
What Lollipop is doing is trying to bring down the lags in your apps and games. For this, it is increasing the time taken to load an app. The idea is to give you better performance while you're using the phone, compromising on the general snappiness of the UI. Whether the trade off is valid is completely a matter of one's own opinion.
Follow Us
Prasid Banerjee
Trying to explain technology to my parents. Failing miserably. View Full Profile
