How do you determine what makes a good OS? What makes iOS vs. Android or Windows Phone vs. BB10, or any other such comparison not just about the fanboyism? Is it even possible to arrive at a scientific conclusion to this question? If we look at entire eco-systems, Android and iOS are obvious choices for buyers because of the sheer amount of apps they have available. However, what’s that got to do with an answer to the question, "What’s the best designed OS out of the box?"
We believe it’s quite possible to compare OSes, based on features, functionality and the implementation of those features (usability). This belief led us to undertake the largest, most complex comparison that the Digit Test Centre has ever undertaken as part of our August 2014 cover story. Hundreds of man hours went into this task, and over a thousand parameters were meticulously checked (for each OS).
Since this is without doubt a controversial topic, we’re going to be a little verbose in explaining what we did, so as to perhaps reduce the hate mail a tad. Bear with us.
The OSes we pitted against each other were:
Now, as most of you know, Android is a very fragmented ecosystem and there are multiple flavours of the OS in various forms available to the user today. This means that users also have various UIs to choose from, and that changes the usability of the phone drastically. Watch this space for a follow-up article that pits all the Android versions against one another as well..
For now though, we’re focusing only on the top four mobile (stock) OSes available in the market.
No matter how used to an OS you think you are, there are hidden settings that you just have never used before. In fact, if you can find the time to go through our entire table you’re probably going to discover a few features you didn’t know your phone had.
Another discovery users of one platform might come across is that the features they thought were unique to the platform they use, are actually pretty universal across all OSes.
As for us, we needed to identify just about every single feature / parameter that the OSes offered out of the box. We included an Android variant (distro? OS? UI? Mod?) – the HTC-made Sense 6.0 (on an HTC One M8) – in order to look for even more features than what just the stock OSes might offer. Better to have more features with NA across the big 4 than to miss out on something. (Plus as we mentioned before, we’re working on a similar comparison for Android variants as well, so we'd have to fill the HTC sheet eventually anyway.)
We noted down each feature in two ways: first a case of yes or no, for whether a feature exists, and then, given that most OSes would offer most features, we noted the amount of steps it would take to use / enable / disable a feature or parameter within the OS.
We also looked at the OS’ capability of allowing the customisation of functionality, but when measuring the number of steps taken to perform a given task, we measured as it was in the stock settings, and not after any customisation we could have done to make the steps shorter. However, we did score the ability to customise, what you can customise, and how easy it is to customise various aspects of the phone.
So for example, let’s take something as simple as increasing the brightness of your screen. There are ways the OS does it out of the box, and then perhaps customisations you can do to make this task faster or easier.
Now, as we mentioned earlier, we’ve ignored the app eco-system (perhaps controversially). We did consider apps that were bundled into the OS by default, but did not score the availability of apps from the respective app stores.
Yes, we agree that apps are integral to the mobile experience, and perhaps are one of the most important considerations you make when making a buying decision, but that has no place in a comparison that aims to find out what is the best designed OS of them all out of the box. At one time Symbian 40 had more apps than Android or iOS, and thus perhaps it’s unfair to consider third-party developed apps as a measure of the OS. A measure of the ecosystem, sure, the OS itself, not really.
Plus, take for example the Android vs iOS fight that fanboys love to indulge in. Both ecosystems have just about all the apps you can imagine, and thus it’s not the apps that will set these two platforms apart. What it comes down to is the way the OS is designed, what functionalities it offers out of the box, how easy or hard is it to find said functionality by a normal human being, etc. That’s exactly what we tested for.
So how do you go about “testing” or “comparing” OSes? For one, we couldn’t run benchmarks, because that’s a test of hardware running an OS and not really a test of the OS itself. What we really needed to do was to break down the functions of a mobile OS into components that highlight a particular use case and rate the OS on its ability to handle that task.
So how do we use our phones the most these days?
Are these the only ways we use our phones? Of course not, some of us may find creative uses for phones, such as using them to read long Tolkien tales, or as a brick to throw at someone (shameless exaggeration), but we’re focussing on the rules here and not the exceptions. As those of you wondering about social media and “sharing”, don’t worry, absolutely every parameter or use case where you would want to “share” something, was looked at under each individual category. So, for example, you want to share a picture you just clicked using your social media accounts, we’ve considered that under photography, etc.
Once we’d decided what we needed to look at (which basically covers over a thousand individual features or settings in each of the OSes), we then proceeded to segregate each and every setting / feature into one of the 10 categories mentioned above.
What sort of parameters you ask? Well, we considered stuff like reassignment of volume keys (to, say, take a photo), Hin-glish support (yep, we’re not kidding, we are Indian, after all) responding to pop-ups, the amount of information on the lock screen (day and date, schedule, installation status, system updates, etc.), setting up VPNs and proxy servers, restricting data usage for mobile hotspots, encryption, fine-grained permissions, system up-time, data logging, adjusting audio balance (for headphones), custom dictionaries, controller support, muting IM/SMS threads, attaching data files to e-mail, merging mail accounts and contacts, scanning business cards, translating web pages, adjusting font sizes and more than a 1000 more such parameters. Here’s the full table for those of you who want a break from reading this wall of text… as much of a break a 1000+ row spreadsheet can offer at any rate.
With so many unique values to be noted, this became a massive exercise that took (what felt like) forever to complete. If that wasn’t hard enough, everything had to be double-checked by someone other than the person who filled in the initial values, and again re-confirmed by a third person whose primary phone was not the platform he was checking data for (to try and do away with blind spots and assumptions), and this last step obviously took the longest.
We noted down every single feature/setting that we could find – collating across OSes to ensure that even features unique to a particular OS found their way into the sheet (eg. BlackBerry OS 10’s unique ability to search through browser history via universal search). This preliminary listing of features was again double-checked by at least two people before we were satisfied with it.
The features/settings were intelligently clumped into groups that were in line with their functionality, so photo-editing features went under “Photography”, and so on.
Each group was then given an overall weight in order of its perceived importance (relying on our own experience testing mobile devices), as were the items that made up that group. This was done before actual scoring was done or results were arrived at in order to minimise bias. So, for example, we arrived at the conclusion that browsing was one of the most important experiences on the phone for all of us way before we knew that Android was winning this parameter with ease.
The presence of a feature/setting as well as the number of steps required to activate or disable said feature/setting was noted down. To keep the test results as consistent as possible, we counted the steps while keeping in mind the default state of the OS in question. For example, you can either pin the settings icon to your Android Home Screen or access it via the quick-access panel, but since the quick-access panel is the default placement of the button, that’s what we considered for the test.
Once we determined whether an OS was able to offer a certain feature, the next step was to find out in which OS was it easiest to turn on or off. The easiest way to do this is to look at how many steps it takes to do something. The touch interfaces today are more than just taps, and gestures are getting a little more complex. A tap would be the easiest gesture, and a swipe in one direction is as easy, but a swipe with more than one finger or a swipe gesture (up and right, for example) is a little more complicated, and other actions such as holding down on a button, etc., are a tad more cumbersome. Besides counting the number of steps needed to accomplish something, we also scored the type of step (tap/swipe/gesture/hold/etc.).
A simple tap or swipe was 1 action, a two-finger swipe was considered to be 1.25 of an action, holding down for a couple of seconds was considered to be 1.5 actions, and so on. Obviously, the OS that required the fewest number of actions to perform a particular task would get the highest score for that feature. We didn’t double score – give an OS points for having a feature and then again give it points for the same feature for offering the easiest / quickest method of setting it. If the feature we were scoring for gave a simple Yes / No answer and steps weren’t required, we scored it 1/0. If it did, we counted the steps needed and scored accordingly (lower is better obviously)
In the course of conducting this comparison, we were caught completely off-guard by:
If nothing else, the most important thing that this test taught us was a very healthy respect for platforms other than the ones we use on a daily basis, and the mammoth effort that obviously went into their development.
In our minds, and, we’re quite sure, in most of your minds as well, Android was the winner even before we began testing. It seemed such a blindingly obvious winner, given that its plethora of apps and the thriving developer community. Unfortunately (for Android), this test is looking at the design of the OS rather than the popularity of the platform and in that regards, stock Android is surprisingly lacking. Perhaps it's a leave it to the apps philosophy, but out of the box it's a little underwhelming when compared to the competition.
Let’s consider universal search for example. One would expect an OS that’s being mentored by Google to have the best overall search experience of them all, but it’s just not as good as you would expect. Both iOS and BB10 do better than Android when it comes to search functionality (internal phone search) and beat it by a large margin. How large? Well, believe it or not, more than double Android’s score!
The default Android Search tool doesn’t search through SMS, calendar entries, E-mail and much more. Those who are used to Android probably won’t even notice it because you didn’t know that it was even possible, but for those migrating from other platforms, especially BlackBerry, this would be a glaring omission.
The same can be said for photography as well. The new “Android Camera” app is a huge disappointment and nowhere near as feature-rich, and in-effect, as functional, as the camera apps on other platforms, BlackBerry and Windows Phone included. Again, remember this is a comparison of the camera software, not hardware, so don't shoot off angry mails about how your camera clicks better photographs than any other platform.
What does it do right? For one thing, Android 4.4.4 features the most feature-rich mobile browser that we’ve had the pleasure of using. While it does lose out on something as convenient as Safari’s Reading List feature, it’s syncing features are incomparable; iOS does manage to come a close second though.
Something else that Android does well is customisability, which was to be expected, but there were issues here as well. Stock Android is simply not as customisable as one would expect. Simple things like customising your quick settings are just not present. You can, of course, just ignore stock Android and jump to CyanogenMod or some other such ROM, but the true customisability of ROMs such as that are something we’ll be looking into later.
Overall, Android manages a distant third, a full 8 points behind our winner and just barely 1.5 points ahead of WIndows Phone in fourth.
To be completely honest, we’ve always respected Apple for what they’ve done with iOS as a platform, but the full extent of the work that they’ve put into the OS was largely ignored by many of us and we really didn’t expect the OS to fare as well as it did. The price for their phones in India kind of turns us off, truth be told.
The big shocker was notifications. iOS didn’t have real notifications a couple of iterations ago (iOS 5) and now all of a sudden we discover that it has the best implementation of a notification system across all platforms. It’s the subtle design decisions that make this possible, the indicator on the icons, location-specific reminders, the ability to allow repeated calls to notify you even when night-mode is enabled and so much more. None of the other OSes even came close to providing such a degree of information and customisability.
Interestingly, the default camera app in iOS is quite awful in terms of manual control. One might argue that the iPhone 5s has an amazing camera and that it probably doesn’t need as many fine-grained settings as the camera app on a rival, but as we've said before, this is about the OS and not the hardware. Plus, what happens when the same camera app is on a lower specced iPod Touch or older iPad? You might want some extra controls on those devices to tweak your pictures.
Search is another area where iOS shines and while it still isn’t as good as BlackBerry’s native search functionality, it was still more than enough to take the second place in our tests with a very healthy score of 75%. Way ahead of Android (35%) and just 10 points behind BB10 (85%).
iOS suffers badly in two departments – managing contacts and customisation. When it comes to contacts, the dialler is very basic and just seems to deal with contacts as an after-thought, unlike the more integrated approach used by just about every other mobile OS. Customisation is another sore point with this OS because it simply doesn’t let you change a lot of the default behaviour.
iOS manages to bag the second spot with its excellent notification system, good browser and excellent search functionality.
The underdog takes the crown. The BlackBerry 10.2 OS is undoubtedly the best designed mobile OS out of the box that’s currently available in the market. We were stunned by the result and to be honest, we delayed sending the magazine to print a day just to recheck everything. BlackBerry’s been behind the times for some time, if we're honest, and BB10 was supposed to be the game-changer. Quite frankly though, most people haven't given the OS much attention. While we’re not arguing the justification for that disdain, it’s quite clear that the OS deserves much more attention than it actually receives.
Now BlackBerry OS 10.2 (10.2.1.2977 for those who want specifics) has, by today’s standards, a very lacklustre browser – lacking even basic sync functionality, a messaging app (SMS) that’s also very basic, and that’s it. Those were the only real downsides to the OS. The browser itself is not terrible, and gets most things done, but with a whole range of other features being offered by the other platforms, somehow quick rendering and a reader mode don't make up the gap.
BlackBerry OS 10.2 boasts of the most feature-rich search engine, an e-mail and contact management system that’s second to none, and is also surprisingly more customisable than we thought it would be, and also offers a good photography and video capture experience.
The sheer functionality offered by the OS, out-of-the-box, just cannot be ignored. Once you complete the initial setup, the device is perfectly capable of performing just about any and every task you would expect from a phone, and then some.
Take the search functionality, it truly is a universal search. A function that scans every bit of data on your phone, including browser history, e-mail and SMS conversations and much more. No other OS in our test was capable of performing these actions. In effect, the search is so effective that BlackBerry OS 10.2 walks away with scores that indicate it to be more than twice as good as Android’s implementation. Only iOS managed to come even close to BB10’s performance in this department.
E-mail was another area where BlackBerry OS 10.2 outshone every other OS (we expected this one). However it was the margin of victory (a score of 83.2, which is a full 30 points ahead of Android’s gmail app) that surprised us a little. The reason for this massive performance difference lies in the way the OS deals with e-mail. While the others are content to leave e-mail management to an app, BlackBerry OS 10.2 integrates e-mail into the OS, making it as native to the platform as a dialler is to a phone. This lends the OS some distinct advantages, particularly when it comes to managing multiple accounts and search, courtesy of the “Hub”.
Anyone who’s used a BlackBerry device knows that it’s a device that places communication first and entertainment second, and that comes across very clearly in our tests. Entertainment options, which would include the browser, photography, etc., are functional components and powerful enough to not give the average user cause for complaint. At the same time, the communication modules are so well designed and integrated into the OS that they far outstrip the competition and place this OS as the clear leader out of the box.
Windows Phone placed last on our test and it’s actually a little difficult to pin-point the problem. The OS is not bad really, it does perform all the actions you would expect an OS to perform in the sense you can send and receive calls and SMS, manage e-mail and even includes a notification drawer with quick access settings.
The real problem, as far as we can judge is that the OS does everything that you expect it to, but it just doesn’t do those things as well or as efficiently as other OSes. It does handle contacts and messaging better than almost every other OS (excluding BB10) courtesy of its People Hub, but that’s the only thing we can really say about it.
In essence, Windows Phone needs more exclusivity and focus than its People Hub to outshine the other three.
What we’ve given you so far is just part one of this quest of comparing OSes and their variants. With refreshes of all of these OSes coming soon, new OSes such as Ubuntu Phone, and even newer Android variants such as on the Amazon Fire Phone, the OnePlus One, Xiaomi Mi 4 and so on there’s still loads of work to be done.
Up next we will be working on the Android variants comparison, so keep an eye out for that.
Also remember to get the August 2014 issue of Digit from here to read the original story.
You can view the full version of our table here.
For those interested in the simpler scoring sheet:
| Score Sheet | ||||
| OS | Android 4.4.4 | iOS 7.1.2 | BlackBerry 10.2 | Windows Phone 8.1 |
| Browser | 66.28 | 61.8 | 40 | 50.47 |
| | 53.07 | 46.6 | 83.2 | 44.93 |
| Messaging | 36.6 | 43.2 | 34.2 | 60.2 |
| Settings and Customisation | 55.48 | 40.09 | 53.21 | 45.13 |
| Contacts and dialler | 51.25 | 40.33 | 62.33 | 58.28 |
| Photography | 43.9 | 47.24 | 55.13 | 54.05 |
| Notifications | 29.4 | 69 | 36.7 | 29.2 |
| Lock Screen | 55.8 | 51.2 | 48.78 | 37.2 |
| Search | 35 | 75.3 | 85.3 | 54.7 |
| Video | 32.72 | 38.6 | 46.76 | 22.56 |
|
|
|
|
|
|
| Final Overall Score | 48.44 | 52.1 | 56.91 | 46.68 |
Remember to send us feedback at editor@digit.in.
Not all of you will want to go through a table with over 1000 rows to try and figure out what we did, and then try and apply your own preferences to our findings. Instead, we’ve come up with an easy way for you to do that.
We’ve done all the difficult scoring. All you need to do is set the weights of 10 different parameters we tested on, make sure it totals to 100, and you will have the final scores adjusted accordingly.
So, for example, if all you care about is browsing and contact management on a phone, set the desired values (say, 60% browsing, 40% contacts, and 0 for everything else) and this calculator will tell you what each OS scores for your preferences in our test. To go back to the weights we assigned to the categories, click the Set Defaults button on the top.
Enjoy!