Generative AI is on Mobile and it’s Powered by Arm
Generative AI, which includes today’s well-known, highly publicized large language models (LLMs), has arrived at the edge on mobile. This means that AI generative inferences, from generating images and videos to understanding words in context, are starting to be processed entirely on the mobile device, rather than being sent to the Cloud and back.
Arm is the foundational technology to enable AI to run everywhere and when it comes to generative AI on mobile, there are some exciting, new developments that demonstrate this in action, from the latest AI-enabled flagship smartphones to LLMs being directly processed on the Arm CPU.
New AI-powered smartphones
High performance AI-enabled smartphones are now on the market, which are built on Arm’s v9 CPU and GPU technologies. These include the new MediaTek Dimensity 9300-powered vivo X100 and X100 Pro smartphones, Samsung Galaxy S24, and the Google Pixel 8.
The combination of performance and efficiency provided by these flagship mobile devices are delivering unprecedented opportunities for AI innovation. In fact, Arm’s own CPU and GPU performance improvements have doubled AI processing capabilities every two years during the past decade.
This trend will only advance in the future with more AI performance, technologies, and features on our robust consumer technology roadmap. This will be supported by the rise of AI inference at the edge, the process of using a trained model like LLMs to power AI-based applications, with CPUs being best placed to serve this need as more AI support and specialized instructions continue to be added.
It all starts on the CPU….
In most cases, the use of AI on our favorite mobile devices starts on the CPU, with some good examples being face, hand and body tracking, advanced camera effects and filters, and segmentation across the many social applications. The CPU will handle such AI workloads in their entirety or be supported by accelerators, including GPUs or NPUs. Arm technology is crucial to enabling these AI workloads, as our CPU designs are pervasive across the SoCs in today’s smartphones used by billions of people worldwide.
This has led to 70 percent of AI in today’s third-party applications running on Arm CPUs, including the latest social, health and camera-based applications and many more. Alongside the pervasiveness of the designs, the flexibility and AI capabilities of the Arm CPU makes it the best technology for mobile developers to target for their applications’ AI workloads.
In terms of flexibility, Arm CPUs can run a wide variety of neural networks in many different data formats. Looking ahead, future Arm CPUs will include more AI capabilities in the instruction set for the benefit of Arm’s industry-leading ecosystem, like the Scalable Matrix Extension (SME) for the Armv9-A architecture. These help the world’s developers deliver improved performance, innovative features and scalability for their AI-based applications.
The combination of leading hardware and software ecosystem support means Arm has a performant compute platform that is enabling the rise of generative AI at the edge, which could include gaming advancements, image enhancements, language translation, text generation and virtual assistants. We will be demonstrating some examples of these next-gen AI workloads and more at Mobile World Congress 2024.
LLM on mobile on the Arm compute platform
We have produced a virtual assistant demo that utilizes Meta’s LLAMA2-7B LLM on mobile via a chat-based application. The generative AI workloads take place entirely at the edge on the mobile device on the Arm CPUs, with no involvement from accelerators. The impressive performance is enabled through a combination of existing CPU instructions for AI, alongside dedicated software optimizations for LLMs through the ubiquitous Arm compute platform that includes the Arm AI software libraries.
As you can see from the video above, there is a very impressive time-to-first token response performance and a text generation rate of just under 10 tokens per second that is faster than the average human reading speed. This is made possible by highly optimized CPU routines in the software library developed by the Arm engineering team that improves time-to-first token by 50 percent and text generation by 20 percent, compared to the native implementation in the LLAMA2-7B LLM.
The Arm CPU also provides the AI developer community with opportunities to experiment with their own techniques to provide further software optimizations that make LLMs smaller, more efficient and faster.
Enabling more efficient, smaller LLMs means more AI processing can take place at the edge. The user benefits from quicker, more responsive AI-based experiences, as well as greater privacy through user data being processed locally on the mobile device. Meanwhile, for the mobile ecosystem, there are lower costs and greater scalability options to enable AI deployment across billions of mobile devices.
Find out more information about this demo from the Arm engineers that developed it in this technical blog.
Driving generative AI on mobile
As the most ubiquitous mobile compute platform and leader in efficient compute, Arm has a responsibility to enable the most efficient and highest-performing generative AI at the edge. We are already demonstrating the impressive performance of LLMs that are running entirely on our leading CPU technologies. However, this is just the start.
Through a combination of smaller, more efficient LLMs, improved performance on mobile devices built on Arm CPUs and innovative software optimizations from our industry-leading ecosystem, generative AI on mobile will continue to proliferate.
Arm is foundational to AI and we will enable AI everywhere, for every developer, with the Arm CPU at the heart of future generative AI innovation on mobile.
Any re-use permitted for informational and non-commercial or personal use only.
Editorial Contact
Latest on Twitter
Congratulations to Tamika Curry Smith, our Chief DEI Officer who has been chosen as one of @SavoyNetwork's Most Influential Executives in Corporate America for 2024. 👏
The honor recognizes Tamika's commitment to community, leadership, and success: https://okt.to/2NvDOZ
This #EarthDay, we're celebrating our collaboration with @UNICEF, to unlock the life-changing potential of technology in Malawi.💧
Our investment has helped UNICEF install two water supply systems, helping provide 11,000 people with access to safe water and sanitation services.
ISA parity is accelerating silicon and software deployment in automotive! 🚗💨
👇 At #ew24 we showcased what this looks like in practice via virtual platforms with our Neoverse-based cloud instances and new Arm AE IP, which brings almost 100% ISA parity. https://okt.to/cjVaFN
The future of automotive is here. 🚀
Hear from Dipti Vachani on @IBDinvestors as she shares details on the game-changing performance and functional safety features of our new Armv9-based AE processors and server-class Neoverse technology: https://okt.to/WVtw58
AI is expected to increase data center power consumption 3️⃣x by 2030.
To offset this, hyperscalers like Amazon and Google turn to Arm Neoverse to reduce power without compromising performance.
That's why the future of AI is built #onArm. https://okt.to/LPVFWd
We're working with our partners @Simprints, @Gavi and @_GHSofficial to enable healthcare equity in Ghana.
Built #onArm, Simprints technology is enabling vaccine equity by making it easier to verify vaccine delivery by individual: https://okt.to/MIcgxu
If Arm technology is in there, then the future will be too.
Our energy-efficient compute platforms are foundational to the world's technology and AI computing, shaping how we work, play and connect.
That's why the future is built #onArm. https://okt.to/M9wqc8
Here's to a fantastic week at Embedded World! 🎉
From lightning talks and speaker sessions to demos of our latest automotive and IoT technologies, we brought our vast ecosystem together to showcase what's possible when you build #onArm.
Thank you to everyone that joined us. 👏
🚗 Software-defined vehicles (SDVs) are the future of automotive and they're being built #onArm.
Robert Day joins @Sonatus and @MotorTrend on the latest podcast to talk all things SDVs and where the industry is headed.
🔊👇 Listen below.
🏆 And the winner of the Embedded Awards 2024 goes to… @Ambiq_AI's Apollo510 MCU!
This winning tech opens the door to new local AI opportunities on billions of devices.
Listen in as the team talk us through their exceptionally efficient Arm-based SoCs at #EmbeddedWorld.
David Maidment recently shared his thoughts on the future of edge computing at #EmbeddedWorld!
Missed the talk? Catch up on his insights into the evolution of edge computing across billions of diverse connected devices to build a secure AI-enabled future: https://okt.to/EtfSRN
AI is everywhere at this year's #EmbeddedWorld.
🚘 With the demand for more autonomy, increasingly advanced user experiences and the trend toward electrification Dennis Laudick shares why AI will remain crucial to drive innovation in the software-defined vehicle.
#ew24
Two identical demos. One different NPU.
Our new Ethos-U85 NPU is optimized for transformer networks to deliver the performance needed for edge AI use cases - including high performance voice and vision experiences.
This makes it perfect for AI applications like Gen AI. #ew24
We're ready for the final day of #EmbeddedWorld! 🚀
It's been a fantastic show so far showcasing our power-efficient compute platforms that are fueling the future of edge AI.
Reply with an emoji to let us know if you'll be visiting us on the show floor. 👋
#ew24
What's trending at #EmbeddedWorld?
AI, AI and more AI. 🔥
@pnwilliamson shares where you might find edge AI in action from our ecosystem partners building the future #onArm. #ew24
At the Arm booth at #EmbeddedWorld you'll find:
✅ Partner demos from the likes of @NXP, @Ambiq_AI and @_Elektrobit
✅ Partner Lightning Talks from @Microsoft, @EdgeImpulse and more
✅ Our latest IoT and Automotive tech
#ew24
🚀 We're taking edge AI innovation to the next level with the new Arm Ethos-U85 NPU and Corstone-320 IoT Reference Design Platform.
@pnwilliamson shares how these new technologies are transforming future voice, audio and vision applications live from #EmbeddedWorld. #ew24
Get to know LaunchX from NetsPresso by @nota_ai, on show at #EmbeddedWorld. 👇
Designed to help you easily convert and benchmark AI models, it supports a ton of Arm-based devices from our partners including Arduino, NXP, Renesas, Raspberry Pi and NVIDIA. #ew24
🇺🇸🇯🇵 We're proud to support new AI research partnerships between universities in the US and Japan.
Through close collaboration between business, academia and government we can continue to advance AI innovation to transform industries and improve lives. https://okt.to/9JbahF
📢 Custom Arm-based silicon from @GoogleCloud is here!
Axion, built on Armv9 Neoverse V2, will bring performance & energy efficiency gains to AI inferencing & general-purpose workloads, like web and app servers.
Congrats to the team! 👏 https://okt.to/xS4cBN #GoogleCloudNext
It's time for #EmbeddedWorld! 🙌
We're in Nuremberg this week showing you how to build your AI-accelerated future #onArm. 🎉
Can’t join us in person? Keep up to date on social media for live updates from the show! #ew24
🆕 Get ready to accelerate edge AI with the new Arm Ethos-U85 NPU and Corstone-320 IoT Reference Design Platform. ⚡
With performance and efficiency uplifts, developers can now deliver greater AI compute in smaller, more cost-sensitive devices: https://okt.to/cJPTCQ
When it comes to AI and ML at the edge, challenges like hardware fragmentation and performance optimization stand in the way of widespread adoption.
@pnwilliamson shares how developers can navigate this, with an invitation to join us at Embedded World: https://okt.to/tpsPTd
With the AI boom fueling demand for compute, our CEO Rene Haas speaks to @jchatterleyCNN on the importance of ecosystem collaboration to unlock unique capabilities we've not seen before.
Listen to the @CNN podcast at 28:38 for the full interview 👉 https://okt.to/mcH1tF
