Unlocking New Real-world Generative AI Use Cases on the Mobile CPU
In 2022, the first example of generative AI emerged through text to image generation in the cloud. The text prompt was “a photograph of an astronaut riding a horse”, with the generative AI workload creating an image of just that. While there were some issues with the image, it showcased the awe-inspiring power and potential of generative AI workloads.
Rather than running this use case in the cloud, I remember thinking to myself at the time “this is great, but could it ever be processed entirely on a mobile device?”
Generative AI is (already) part of today’s smartphone experience
Fast-forward to today and you can. In fact, many generative AI workloads, like image generation and text summarization, that are now a common part of the modern smartphone experience are being processed at the edge – on the device. This is largely thanks to the computing capabilities of today’s AI-enabled flagship smartphones and the large language models (LLMs) behind generative AI becoming smaller and more efficient. And these trends will continue to evolve, with generative AI set to be a part of every single mobile application in the near future.
AI workloads start on the CPU
As we’ve talked about previously, AI on mobile starts on the CPU. It offers software flexibility and programmability for the world’s developers. On top of this, the ubiquity of the CPU, which features in every single digital consumer device on the planet, means developers can “write once, deploy everywhere” when creating their applications, ensuring they reach the widest number of users.
Earlier this year, we demonstrated a chatbot demo deployed to be a virtual teaching assistant for science and coding that runs on mobile on the CPU. The success of the demo meant we started exploring other practical generative AI mobile use cases that run on the Arm CPU and could be used every day by the average smartphone user. This led to the creation of three new demos – group chat summarization, voice note summarization and a real-time voice assistant. Like the chatbot demo, these process and run generative AI workloads entirely on the device, which provides privacy, latency and cost benefits compared to sending the data to the cloud to be processed.
The new generative AI demos
For me personally, group chat and voice note summarization are brilliant life hacks. Like most smartphone users, I can get inundated with various messages and voice notes from friends and family, so being able to use generative AI to summarize what’s been said is invaluable.
The group chat summarization demo quickly distills group chat messages with multiple participants down to the key points in an easily digestible format. Even though the demo itself is showcasing group chat messages, it can be used for other applications like summarizing emails. This use case could also be multimodal and even include pictures as part of the summarization.
The voice note summarization demo shows how an LLM and speech-to-text model can work together in a pipeline to summarize and transcribe voice notes sent to users, with the model converting the voice note to text and the LLM then summarizing the text. For me personally, this demo is a real time-saver!
Like the previous chatbot demo, what’s innovative about the real-time voice assistant demo is that it’s happening completely in flight mode. This shows the capabilities of the Arm CPU to process generative AI workloads entirely on the device.
The demo uses whisper.cpp for automatic speech recognition before going to the LLM module which uses Google AI Edge’s MediaPipe to run the Gemma 2B model. There is also the option to use the Llama 3 model too. Even using a 3B parameter model, this can achieve a real-time conversation, with a realistic voice and also context retention.
In both the whisper block for speech recognition and LLM block for generating the response, Arm KleidiAI, a collection of highly optimized AI kernels that deliver high performance for generative AI workloads, is integrated to enhance performance. This use case could also be used across automotive applications for hands-free driver to device interactions user voice command, such as asking for directions while driving, or interactive dialogue with characters in gaming.
For all three demos, we use AI-enabled flagship smartphones that adopt Armv9 CPU technologies in their chipsets, including Google Pixel 8 and Pixel 8 Pro (Google Tensor G3 chipset), Xiaomi Redmi K60 Ultra and vivo X100 (MediaTek Dimensity 9300 chipset). The Armv9 CPU technologies integrate the latest architecture features for enhanced AI performance, including SVE2.
In the future, AI-enabled flagship smartphones built on Arm CPUs will utilize the Scalable Matrix Extension (SME) architectural feature, which accelerates AI workloads and enables improved performance, power efficiency and flexibility for AI-based applications running on the Arm CPU.
Looking to the future
While present day possibilities from generative AI are incredible, the future is likely to be even more exciting. In fact, I believe that we are scratching the surface of generative AI on mobile, particularly with image and video generation.
Recently, OpenAI showcased text to video generation, and Luna Labs demoed image to video generation. While both generative AI workloads are being processed in the cloud, if we follow the current trajectory, then there is no reason why they couldn’t be processed on mobile on the CPU in two years’ time – just like the astronaut riding the horse!
Generative AI on mobile runs on Arm
With so many different use cases and workloads that are possible, generative AI is consolidating the smartphone as the center of personal and professional compute. This makes it a hugely exciting time for generative AI in the mobile space.
Through our ubiquitous CPU technologies that feature in 99 percent of the world’s smartphones and industry-leading mobile ecosystem, Arm is the company which is enabling these amazing possibilities.
As we continue to add more capabilities and architecture features to the Arm CPU, alongside unlocking yet more AI performance for developers through Arm Kleidi, Arm is the mobile platform for the future of AI.
AI runs on Arm
Learn about how Arm is accelerating AI everywhere, from cloud to edge.
Any re-use permitted for informational and non-commercial or personal use only.
Editorial Contact
Latest on X
The combined power and performance of Arm and @GitHub is optimizing the developer experience and unlocking advances in AI.
Here's to many more years of partnership and building the future of AI #onArm. 💪
From versatility and cost-effectiveness to scalability and power efficiency, here's 5 key reasons why CPUs are the top choice for AI ⏬
Top 5 Reasons why CPU is the Best Processor for AI Inference - Embedded
Advanced artificial intelligence (AI), like generative AI, is enhancing all our smart devices. However, a common misco...
www.embedded.com
From on-CPU AI processing to chiplet-enabled designs, hyperscalers get more control, efficiency, and power to meet the demands of AI when building #onArm.
@TheNextPlatform shares how custom Arm CPUs are transforming the data center.
Custom Arm CPUs Drive Many Different AI Approaches In The Datacenter
Sponsored Feature Arm is starting to fulfill its promise of transforming the nature of compute in the datacenter, ...
www.nextplatform.com
Arm founding CEO & former chairman, Sir Robin Saxby, has been inducted into the City of London Engineering Hall of Fame!
This acknowledges the transformational impact of Saxby's work on the Arm compute platform, laying the foundation for who we are today: https://newsroom.arm.com/blog/sir-robin-saxby-city-of-london-engineering-hall-of-fame?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk29_exec-comms_na
We're entering a new partnership with @Panasonic Automotive Systems.
To accelerate development cycles in the automotive industry, we'll adopt and extend VirtIO, the device virtualization framework that will help standardize vehicle architecture for SDVs: https://newsroom.arm.com/news/arm-and-panasonic-partner-to-standardize-sdvs?utm_source=twitter&utm_medium=social-organic&utm_content=newsletter&utm_campaign=mk12_automotive_na
We're delighted to welcome Charlotte Eaton back to Arm. 🎉
Charlotte rejoins us as Chief People Officer, drawing on her wealth of experience at Arm and in other leadership roles to empower our people to do the best work of their careers: https://newsroom.arm.com/news/arm-announces-appointment-of-charlotte-eaton-as-chief-people-officer?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk29_exec-comms_na
Demand for the Arm compute platform remains strong in the age of AI.
In Q2 FYE25, we achieved record royalty revenue for Q2, with strong YoY growth thanks to the increased demand for energy-efficient compute and more complex AI chips built #onArm:
Frederique Olivier relies on Arm to capture the beauty of the natural world. 🌎
With the help of Arm, she can shoot images with better quality and source power in extreme conditions.
Discover the world from Frederique's eyes, made possible #onArm: https://newsroom.arm.com/blog/purpose-on-arm-frederique-olivier-photographer?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk08_brand_na
🗓️ Join us on Wednesday November 6 for our next financial results conference call.
After market close, we'll report our earnings for the second quarter of fiscal year 2025, followed by a conference call to review our results and business outlook: https://newsroom.arm.com/news/arm-announces-earnings-release-date-for-second-quarter-fiscal-year-ended-2025?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk29_exec-comms_na
6G is on the horizon, promising ultra-fast speeds, low latency, and seamless digital and physical experiences.
Learn what it takes to bring new AI capabilities to life on 6G: https://newsroom.arm.com/blog/arm-and-alphawave-semi-unite-on-6g-chiplets?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk03_infrastructure_na
Experience new levels of performance and efficiency with the general availability of @GoogleCloud C4A VMs!
C4A features the Arm Neoverse V2-powered Google Axion CPU to support the most demanding AI & general purpose workloads: https://cloud.google.com/blog/products/compute/try-c4a-the-first-google-axion-processor
Congrats @GoogleCloud! 👏
Catch the full conversation to hear why suggesting AI is overhyped "couldn't be further from the truth": https://www.bloomberg.com/news/videos/2024-10-22/arm-ceo-dismisses-ai-bubble-fears-video
#BloombergTech
"AI is going to run on Arm. Period."
Arm is the only company on the planet that can run workloads of any size. There's plenty of opportunity to grow and drive innovation in the space, and we're leading this.
Thanks to #BloombergTech for having us!
At #GitHubUniverse or watching the live stream?
➡️Learn how to maximize performance with GitHub Copilot and GitHub Actions. Catch the team on the Main Stage at 3:45pm PT!
➡️Attending in person? Join our AI roundtable later today - spaces are limited! https://www.arm.com/resources/registration/github-universe-roundtable
Get ready to streamline development with the Arm extension for @GitHub Copilot!
Coming 🔜, the extension will empower developers to build and test software quickly and efficiently on the Arm compute platform, supporting seamless cloud-to-edge deployment: https://newsroom.arm.com/news/ai-driven-tools-on-github?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk24_developer_na
Ready to future-proof your workloads?
Boost performance and cut costs by migrating to Arm. 💪
Companies like @HoneycombIo and @FusionAuth saw major gains with AWS Graviton and so can you. Explore how and access our migration guide to get started: https://newsroom.arm.com/blog/why-developers-are-migrating-to-arm?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk24_developer_cloud-dev
Panasonic Automotive Systems joined us to discuss how AI is revolutionizing the in-vehicle experience in our latest Viewpoints podcast episode.
Discover how AI is shaping the future of driving, making it safer, more intuitive and more comfortable: https://newsroom.arm.com/podcasts/ai-in-the-car?utm_source=twitter&utm_medium=social-organic&utm_content=podcast&utm_campaign=mk12_automotive_na
The Armv9 Cortex-X925 CPU is setting new standards with a 15% IPC improvement!
Get ready for unparalleled performance and efficiency across flagship AI smartphones and PCs. More details: https://newsroom.arm.com/blog/armv9-cortex-x925-cpu-performance?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk04_client_na
For @NVIDIA, transparency is the key to success. 🔑
To build one of the largest companies in the world, NVIDIA CEO Jensen Huang sought to build a porous organization that empowers people to experiment and improve.
Listen to the full Tech Unheard podcast: https://newsroom.arm.com/podcasts/tech-unheard-episode-one-jensen-huang?utm_source=twitter&utm_medium=social-organic&utm_content=podcast&utm_campaign=mk08_brand_na
The Arm CPU architecture is enabling quicker AI experiences with enhanced security, unlocking new possibilities for AI workloads at the edge.
Find out why the CPU is front and center in the future of AI on The @Verge.
Why AI starts on the CPU
The rise of AI inference means more AI workloads are being processed at the edge.
www.theverge.com
It's a big day for small models!
With @AIatMeta's new quantized Llama models and the Beta release of ExecuTorch, it's now quicker for 20 million Arm developers to create and deploy performant, intelligent gen AI at the edge.
👇We've put it to the test: https://newsroom.arm.com/news/accelerating-edge-ai-with-executorch?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk13_tech-strat_na
We're excited to see the new quantized Llama 3.2 1B and 3B models available on the Arm compute platform.
👉 Developers, get ready to seamlessly integrate these new models into your applications #onArm with no additional modifications or optimizations, saving time and resources
Our collaboration with @vivo_Global and @MediaTek is redefining the AI smartphone. 📱
🤝 Together, we've delivered advanced performance and efficiency to vivo's X200 series with MediaTek's Dimensity 9400 with the latest Armv9 CPUs and Immortalis GPU: https://newsroom.arm.com/news/powering-next-generation-ai-smartphones-mediatek-vivo?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk04_client_na
Collaboration is key to enabling the AI-enabled vehicle. 🚗
At #REAutoUSA, Dipti Vachani, our SVP and GM for Automotive shared how we're working across the stack to deliver solutions that enable software development from day 1, enabled by standards driven by SOAFEE.
