Hyped-up technological phenomena are quite common and most of them end up being a bubble. One that pops without mercy and takes out most of the companies in that domain. The surviving few are the ones who’ve truly innovated or offer a little more than just a buzzword. AI is no different. There are several key players whose names we all know for they’ve become synonymous with AI, for eg. ChatGPT. It’s a Large Language Model (LLM) based chatbot and the most popular of all AI services currently available to the public. And there are many like it but most people will only recall ChatGPT.
Also read: Intel AI Inside: Intel AI hits the road with 12 cores driving your car’s future
ChatGPT, basically, consists of a weighted AI model that’s running on a server somewhere with access to petabytes of information which it can access at the drop of a hat. For anyone who’s used ChatGPT, they’ll be all too familiar with the time taken for each response. That’s because it takes time for the model to run through all the information that it has and generate an adequate response. And the reason why it can give you a response as fast as it currently does is because the model is running on some of the most powerful hardware that money can buy. So does that mean that AI will forever remain on the cloud? Not quite.
Smarter, AI-powered chip
AI requires a lot of horsepower for the training stage i.e. when it is learning. Once deemed to be worthy of being put to use, the AI model no longer requires powerful hardware, it can make do with a fraction of the processing power. This stage is termed as inference. So once an AI is trained, it can function on a relatively low-power processor. You can opt for a more energy-efficient application by using a processor that is designed for AI applications. Say, something like your average smartphone. And if you haven’t realised, most smartphones have already incorporated some form of AI processor for a few generations. A lot of companies call this specialised processor as the Neural Processing Unit (NPU). And unlike the cloud-reliant AI systems of the past, on-device AI brings the power of artificial intelligence directly into the hands of users, offering a blend of personalisation, security, and efficiency previously unimaginable. Previous iterations of the NPU have been quite subtle with very small AI models running off of it for purposes of polishing up that photo that you just shot with your phone camera. With the advent of the AI arms race, all smartphone hardware manufacturers have ramped up efforts significantly allowing for much more capable models to operate directly on your personal device.
Tracing back the lineage of on-device AI, we uncover a history rich with innovation and breakthroughs. From the initial, simple algorithms to today’s sophisticated models capable of running directly on devices, the journey has been nothing short of remarkable. Take for example the recent announcements by Samsung and Google. Their new devices can actually translate real-time phone calls in completely different languages, ramp up image and video quality, transcribe conversation and generate minutes of the meeting and manipulate photos and videos like a movie studio from the ‘80s. I say ‘80s because there are a lot of funny artefacts produced when you try out these features. While Google has its own arm-based processor for its Pixel devices the other device manufacturers have to rely on the industry giants to bring the power of AI on their devices. Qualcomm, a frontrunner in this revolution, has been instrumental in pushing the boundaries of what’s possible, pioneering the development of sub-10 billion parameter models that bring generative AI to devices small and large without compromising on performance. This evolution underscores a significant trend: the movement towards making AI ubiquitous, yet unobtrusive, in our daily lives.
Today, on-device AI is not confined to the realm of smartphones and tablets, those are just the most visible applications. Its footprint extends across a diverse array of form factors, including wearables, automotive systems, and IoT devices. This widespread adoption has been made possible by leaps in AI algorithms and hardware optimisation, allowing even the most compact devices to boast AI capabilities. This democratisation of AI technology signifies a future where intelligence is embedded in every digital interaction, from the way we drive to how we live and communicate.
The perks
The benefits of on-device AI extend far beyond mere convenience. Enhanced privacy, reduced latency, and offline functionality stand out as hallmark benefits. By processing data locally, these AI models ensure that sensitive information never leaves the device, offering a fortress of privacy in an era of increasing data breaches. Moreover, the elimination of cloud dependency means instant responses to user commands, transforming user experience into something seamless and intuitive. As it stands right now, instant responses are a far cry since we’re looking at the first iterations of AI models that directly respond to the user. The term ‘near-instant’ is more appropriate. Nevertheless, these benefits are not just theoretical; they manifest tangibly across all devices, making technology not just faster and safer, but also more aligned with human needs.
True democratisation cannot happen with proprietary AI models. Every manufacturer wants to keep their AI refinements close to their hearts. The rise of open-source AI models is going to be the true game-changer in the democratisation of AI technology. By making cutting-edge models accessible to all, they lower the barrier to entry for developers and small companies, sparking a wave of innovation that was previously confined to tech giants. This open ecosystem fosters a collaborative environment where advancements are shared, not hoarded, propelling the entire industry forward and ensuring that the benefits of AI are widespread and equitable.
The challenges
Despite its promise, the road to ubiquitous on-device AI is fraught with challenges. Hardware limitations, energy consumption, and ethical dilemmas such as bias and privacy concerns are significant hurdles. Not a day goes by without a news article showcasing how certain AI models used copyrighted information to train their models. In fact, several major corporations are holding back on the mass adoption of popular AI models for this very concern. You wouldn’t want to issue a recall or patch out an AI feature because you had later discovered that it was trained on data obtained illegally. Especially, not when you’ve charged a premium for the ‘AI features’.
Industry giants such as Google, Qualcomm, Samsung, and OpenAI are leading several initiatives in this domain, including collaboration with industry partners and investment in ethical AI research highlighting a comprehensive approach to overcoming these obstacles, ensuring that on-device AI can scale responsibly and sustainably. You’ll still come across the occasional article or two stating that even these biggies have inadvertently used data that shouldn’t have been but such instances will go down especially since artists have started fighting back with poisoning tools that embed tiny pieces of data that mess up the AI models which train on them.
Where else will you see on-device AI?
From healthcare monitoring via wearable devices to smart home automation and real-time language translation on smartphones, the applications of on-device AI are as diverse as they are transformative. Not everyone’s bodies are the same. The polymorphism of certain genes across races among human beings results in different responses to certain medicines. So healthcare is one of the domains which has a huge potential for personalisation. Your wearables are going to become much smarter in the coming years to enable personalised care.
While self-driving cars have been a reality for a while now, not all cars come with the same smarts. That’s changing thanks to the Advanced driver-assistance systems (ADAS) that companies such as Qualcomm have been pioneering. Real-time decision-making, automatic braking, maintaining lane discipline and detecting pedestrians are just some of the smart features creeping into premium cars. It won’t be long before they’re present even in the cheapest cars that you can buy. After all, the hardware enabling these services isn’t that cost-prohibitive anymore.
Smart home devices such as thermostats, lights, and security cameras have also started incorporating AI models, albeit not as smart as the ones on our phones. Portable diagnostic devices are also seeing AI models being built into them. Patients with chronic conditions can be alerted way in advance and ailments can be detected way before they become chronic. Our life expectancies can only go up with such smart health monitoring devices. Whether it’s providing critical health insights on the go or breaking down language barriers in real time, on-device AI is reshaping our world in profound ways.
On-device AI is the new normal
The impact of on-device AI extends far beyond mere technological innovation; it heralds a new chapter in our relationship with technology – a new normal where devices act as extensions of ourselves, understanding and anticipating our needs while safeguarding our privacy. As we stand on the brink of this new era, it’s clear that on-device AI is not just transforming our devices; it’s reshaping our world, making technology more personal, intelligent, and, ultimately, human.
Follow Us
Mithun Mohandas
Mithun Mohandas is an Indian technology journalist with 10 years of experience covering consumer technology. He is currently employed at Digit in the capacity of a Managing Editor. Mithun has a background in Computer Engineering and was an active member of the IEEE during his college days. He has a penchant for digging deep into unravelling what makes a device tick. If there's a transistor in it, Mithun's probably going to rip it apart till he finds it. At Digit, he covers processors, graphics cards, storage media, displays and networking devices aside from anything developer related. As an avid PC gamer, he prefers RTS and FPS titles, and can be quite competitive in a race to the finish line. He only gets consoles for the exclusives. He can be seen playing Valorant, World of Tanks, HITMAN and the occasional Age of Empires or being the voice behind hundreds of Digit videos. View Full Profile
