Arm Ethos-U85 NPU: Unlocking Generative AI at the Edge with Small Language Models
As artificial intelligence evolves, there is increasing excitement about executing AI workloads on embedded devices using small language models (SLM).
Arm’s recent demo, inspired by Microsoft’s “Tiny Stories” paper and Andrej Karpathy’s TinyLlama2 project, where a small language model trained on 21 million stories generates text, showcases endpoint AI’s potential for IoT and edge computing. In the demo, a user inputs a sentence, and the system generates an extended children’s story based on it.
Our demo featured Arm’s Ethos-U85 NPU (Neural Processing Unit) running a small language model on embedded hardware. While large language models (LLMs) are more widely known, there is growing interest in small language models due to their ability to deliver solid performance with significantly fewer resources and lower costs, making them easier and cheaper to train.
Implementing A Transformer-based Small Language Model on Embedded Hardware
Our demo showcased the Arm Ethos-U85 as a small, low-power platform capable of running generative AI, highlighting that small language models can perform well within narrow domains. Although TinyLlama2 models are simpler than the larger models from companies like Meta, they are ideal for showcasing the U85’s AI capabilities. This makes them a great fit for endpoint AI workloads.
Developing the demo involved significant modeling efforts, including the creation of a fully integer int8 (and int8x16) Tiny Llama2 model, which was converted to a fixed-shape TensorFlow Lite format suitable for the Ethos-U85’s constraints.
Our quantization approach has shown that fully integer language models can successfully balance the tradeoff between maintaining strong accuracy and output quality. By quantizing activation, normalization functions, and matrix multiplications, we eliminated the need for floating-point computations, which are more costly in terms of silicon area and energy—key concerns for constrained embedded devices.
The Ethos-U85 ran a language model on an FPGA platform at only 32 MHz, achieving text generation speeds of 7.5 to 8 tokens per second—matching human reading speed—while using just a quarter of its compute capacity. In a real system-on-chip (SoC), performance could be up to ten times faster, significantly enhancing speed and energy efficiency for AI processing at the edge.
The children’s story-generation feature used an open-source version of Llama2, running the demo on TFLite Micro with an Ethos-NPU back-end. Most of the inference logic was written in C++ at the application level. Adjusting the context window enhanced narrative coherence, ensuring smooth, AI-driven storytelling.
The team’s adaptation of the Llama2 model to run efficiently on the Ethos-U85 NPU required careful consideration of performance and accuracy due to the hardware limitations. Using mixed int8 and int16 quantization demonstrates the potential of fully integer models, encouraging the AI community to optimize generative models for edge devices and expand neural network accessibility on power-efficient platforms like the Ethos-U85.
Showcasing the Power of the Arm Ethos-U85
Scalable from 128 to 2048 MAC units (multiply-accumulate units), the Ethos-U85 achieves a 20% power efficiency improvement over its predecessor, the Ethos-U65. A standout feature of the Ethos-U85 is its native support for transformer networks, which earlier versions could not support.
The Ethos-U85 enables seamless migration for partners using previous Ethos-U NPUs, allowing them to capitalize on existing investments in Arm-based machine learning tools. Developers are increasingly adopting the Ethos-U85 for its power efficiency and high performance.
The Ethos-U85 can reach 4 TOPS (trillions of operations per second) with a 2048 MAC configuration in silicon. In the demo, however, a smaller configuration of 512 MACs on an FPGA was used to run the Tiny Llama2 small language model with 15 million parameters at just 32 MHz.
This capability highlights the potential for embedding AI directly into devices. The Ethos-U85 effectively handles such workloads even with limited memory (320 KB of SRAM for caching and 32 MB for storage), paving the way for small language models and other AI applications to thrive in deeply embedded systems.
Bringing Generative AI to Embedded Devices
Developers need better tools to navigate the complexities of AI at the edge, and Arm is addressing this with the Ethos-U85 and its support for transformer-based models. As edge AI becomes more prominent in embedded applications, the Ethos-U85 is enabling new use cases, from language models to advanced vision tasks.
The Ethos-U85 NPU delivers the performance and power efficiency required for innovative, cutting-edge solutions. Like the “Tiny Stories” paper, our demo represents a significant advancement in bringing generative AI to embedded devices, demonstrating the ease of deploying small language models on the Arm platform.
Arm is opening new possibilities for Edge AI across a wide range of applications, positioning the Ethos-U85 to power the next generation of intelligent, low-power devices.
Read how Arm is accelerating real-time processing for edge AI applications in IoT with ExecuTorch.
Any re-use permitted for informational and non-commercial or personal use only.
Editorial Contact
Latest on X
This is what a day in the life looks like for Graduate Product Manager, Conall Farney. 📹
Since joining us as a Sales Intern, Conall moved into a new role matching his growing passions.
Discover your path with us: https://okt.to/KdWshB
Game-changing quantum breakthrough: Equal1 runs Arm Cortex at 3.3 Kelvin (-269.85°C)!
Equal1's innovative approach could make quantum computing as accessible as traditional servers.
From drug discovery to AI, the future is getting cooler ❄️🌡️ https://newsroom.arm.com/blog/pushing-quantum-computing-boundaries-with-arm-equal1?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk13_tech-strat_na
Here at Arm, we know that prioritizing your wellbeing is key to unlocking your potential.
👉 That’s why we’re committed to nurturing our team’s passions to drive innovation. Scroll through to hear what some of our people have to say about how Arm helps them thrive.
When software architecture isn't standardized, hardware changes can cause issues.
@SchneiderUKI collaborated with us to solve this challenge by bringing in Arm SystemReady, where new hardware works with the software:
SystemReady Implementation at Schneider Electric: Accelerating Development and Enhancing Platform...
How Schneider Electric leveraged Arm SystemReady to accelerate development, enhance platform flexibility and overcome supp...
newsroom.arm.com
Game-changing quantum breakthrough: Equal1 runs Arm Cortex at 3.3 Kelvin (-269.85°C)!
Equal1's innovative approach could make quantum computing as accessible as traditional servers. From drug discovery to AI, the future is getting cooler 🌡️❄️ https://newsroom.arm.com/blog/pushing-quantum-computing-boundaries-with-arm-equal1?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk13_tech-strat_na
AI is causing a fundamental shift in data center infrastructure.
With the flexibility of the Arm compute platform and our relentless focus on energy efficiency and customization, we're at the center of this transformation. Here's why 👇
Arm in the Data Center: The Dawn of a New Era - Creative Strategies
Introduction: 2024 - The Tipping Point for Arm's Data Center Momentum As we close out 2024, the semiconductor indu...
creativestrategies.com
The combined power and performance of Arm and @GitHub is optimizing the developer experience and unlocking advances in AI.
Here's to many more years of partnership and building the future of AI #onArm. 💪
From versatility and cost-effectiveness to scalability and power efficiency, here's 5 key reasons why CPUs are the top choice for AI ⏬
Top 5 Reasons why CPU is the Best Processor for AI Inference - Embedded
Advanced artificial intelligence (AI), like generative AI, is enhancing all our smart devices. However, a common misco...
www.embedded.com
From on-CPU AI processing to chiplet-enabled designs, hyperscalers get more control, efficiency, and power to meet the demands of AI when building #onArm.
@TheNextPlatform shares how custom Arm CPUs are transforming the data center.
Custom Arm CPUs Drive Many Different AI Approaches In The Datacenter
Sponsored Feature Arm is starting to fulfill its promise of transforming the nature of compute in the datacenter, ...
www.nextplatform.com
Arm founding CEO & former chairman, Sir Robin Saxby, has been inducted into the City of London Engineering Hall of Fame!
This acknowledges the transformational impact of Saxby's work on the Arm compute platform, laying the foundation for who we are today: https://newsroom.arm.com/blog/sir-robin-saxby-city-of-london-engineering-hall-of-fame?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk29_exec-comms_na
We're entering a new partnership with @Panasonic Automotive Systems.
To accelerate development cycles in the automotive industry, we'll adopt and extend VirtIO, the device virtualization framework that will help standardize vehicle architecture for SDVs: https://newsroom.arm.com/news/arm-and-panasonic-partner-to-standardize-sdvs?utm_source=twitter&utm_medium=social-organic&utm_content=newsletter&utm_campaign=mk12_automotive_na
We're delighted to welcome Charlotte Eaton back to Arm. 🎉
Charlotte rejoins us as Chief People Officer, drawing on her wealth of experience at Arm and in other leadership roles to empower our people to do the best work of their careers: https://newsroom.arm.com/news/arm-announces-appointment-of-charlotte-eaton-as-chief-people-officer?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk29_exec-comms_na
Demand for the Arm compute platform remains strong in the age of AI.
In Q2 FYE25, we achieved record royalty revenue for Q2, with strong YoY growth thanks to the increased demand for energy-efficient compute and more complex AI chips built #onArm:
Frederique Olivier relies on Arm to capture the beauty of the natural world. 🌎
With the help of Arm, she can shoot images with better quality and source power in extreme conditions.
Discover the world from Frederique's eyes, made possible #onArm: https://newsroom.arm.com/blog/purpose-on-arm-frederique-olivier-photographer?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk08_brand_na
🗓️ Join us on Wednesday November 6 for our next financial results conference call.
After market close, we'll report our earnings for the second quarter of fiscal year 2025, followed by a conference call to review our results and business outlook: https://newsroom.arm.com/news/arm-announces-earnings-release-date-for-second-quarter-fiscal-year-ended-2025?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk29_exec-comms_na
6G is on the horizon, promising ultra-fast speeds, low latency, and seamless digital and physical experiences.
Learn what it takes to bring new AI capabilities to life on 6G: https://newsroom.arm.com/blog/arm-and-alphawave-semi-unite-on-6g-chiplets?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk03_infrastructure_na
Experience new levels of performance and efficiency with the general availability of @GoogleCloud C4A VMs!
C4A features the Arm Neoverse V2-powered Google Axion CPU to support the most demanding AI & general purpose workloads: https://cloud.google.com/blog/products/compute/try-c4a-the-first-google-axion-processor
Congrats @GoogleCloud! 👏
Catch the full conversation to hear why suggesting AI is overhyped "couldn't be further from the truth": https://www.bloomberg.com/news/videos/2024-10-22/arm-ceo-dismisses-ai-bubble-fears-video
#BloombergTech
"AI is going to run on Arm. Period."
Arm is the only company on the planet that can run workloads of any size. There's plenty of opportunity to grow and drive innovation in the space, and we're leading this.
Thanks to #BloombergTech for having us!
At #GitHubUniverse or watching the live stream?
➡️Learn how to maximize performance with GitHub Copilot and GitHub Actions. Catch the team on the Main Stage at 3:45pm PT!
➡️Attending in person? Join our AI roundtable later today - spaces are limited! https://www.arm.com/resources/registration/github-universe-roundtable
Get ready to streamline development with the Arm extension for @GitHub Copilot!
Coming 🔜, the extension will empower developers to build and test software quickly and efficiently on the Arm compute platform, supporting seamless cloud-to-edge deployment: https://newsroom.arm.com/news/ai-driven-tools-on-github?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk24_developer_na
Ready to future-proof your workloads?
Boost performance and cut costs by migrating to Arm. 💪
Companies like @HoneycombIo and @FusionAuth saw major gains with AWS Graviton and so can you. Explore how and access our migration guide to get started: https://newsroom.arm.com/blog/why-developers-are-migrating-to-arm?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk24_developer_cloud-dev
Panasonic Automotive Systems joined us to discuss how AI is revolutionizing the in-vehicle experience in our latest Viewpoints podcast episode.
Discover how AI is shaping the future of driving, making it safer, more intuitive and more comfortable: https://newsroom.arm.com/podcasts/ai-in-the-car?utm_source=twitter&utm_medium=social-organic&utm_content=podcast&utm_campaign=mk12_automotive_na
The Armv9 Cortex-X925 CPU is setting new standards with a 15% IPC improvement!
Get ready for unparalleled performance and efficiency across flagship AI smartphones and PCs. More details: https://newsroom.arm.com/blog/armv9-cortex-x925-cpu-performance?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk04_client_na
