Accelerating Generative AI at the Edge on Arm with ExecuTorch Beta Release
News highlights:
- Combination of Arm Compute Platform with ExecuTorch framework is enabling smaller, optimized models for faster generative AI at the edge
- New quantized Llama models ideal for on-device and edge AI applications on Arm, providing reduced memory footprint and improved accuracy, performance and portability
- 20 million Arm developers can create and deploy more intelligent AI-based applications quicker at scale across billions of edge devices
To realize the true potential of AI, we need to make it accessible to the broadest range of devices and developers. Through collaborating with the PyTorch team at Meta on the new ExecuTorch Beta release, we are fulfilling this mission, bringing AI and machine learning (ML) capabilities to billions of edge devices, as well as millions of developers worldwide.
Generative AI improvements on the Arm Compute Platform with ExecuTorch and new quantized Llama 3.2 1B and Llama 3.2 3B models
The powerful combination of the ubiquitous Arm compute platform, which powers many of the world’s edge devices, with ExecuTorch, a PyTorch-native framework designed to deploy AI models on mobile and edge devices, is enabling smaller, optimized models, including the new quantized Llama 3.2 1B and 3B models. The new quantized models are ideal for generative AI use cases on smaller devices, such as virtual chatbots, text summarization and AI assistants, as they offer a reduced memory footprint, higher levels of accuracy, greater performance and greater portability.
Developers can seamlessly integrate the new quantized models into their applications with no additional modifications or optimizations, saving time and resources. This empowers them to quickly create and deploy more intelligent AI-based applications at scale across a broad range of Arm-powered devices.
As with the new Llama 3.2 large language model (LLM) releases, Arm is optimizing AI performance through the ExecuTorch framework, with this leading to real-world generative AI workloads running faster on edge devices that are built on the Arm Compute Platform. Developers are then able to access these enhancements from day one of the ExecuTorch Beta release.
Accelerating generative AI on mobile with KleidiAI integration
In mobile, Arm’s work with ExecuTorch means virtual chatbots, text generation and summarization and real-time voice and virtual assistants are all running at improved performance entirely on the device on the Arm CPU. This was achieved through integrating KleidiAI, which now introduces micro-kernels optimized for 4-bit quantization, into ExecuTorch via XNNPACK to seamlessly speed-up the execution of AI workloads when running LLMs with 4-bit quantization on Arm Compute Platform. For example, the execution of the prefill stage of the quantized Llama 3.2 1B model will now run 20 percent faster with the KleidiAI integration, leading to speeds of over 400 tokens per second for text generation on some Arm-based mobile devices. This means the end-user will benefit from quicker, more responsive AI-based experiences on their mobile devices.
Learn more about Arm support for ExecuTorch in mobile markets in this blog.
Accelerating real-time processing for edge AI applications in IoT
Meanwhile, in IoT markets, the ExecuTorch work will improve real-time processing for edge AI applications across a broad range of IoT devices, from smart home appliances and wearables to autonomous systems used in retail in industrial IoT. Being able to process more real-time AI tasks on the edge means IoT devices and applications can respond to their environments in milliseconds, which is crucial for safety and functionality.
ExecuTorch can be leveraged across Arm’s Cortex-A CPUs and Ethos-U NPUs to accelerate the development and deployment of edge AI applications. In fact, through combining ExecuTorch with the Arm Corstone-320 reference platform (which is also available as an emulated Fixed Virtual Platform), the Arm Ethos-U85 NPU driver and compiler support into one package, developers can start creating their edge AI applications months before platforms arrive on the market.
Learn more about Arm support for ExecuTorch in IoT markets in this blog.
More accessible, faster edge AI experiences
We believe that ExecuTorch has the potential to be one of the world’s most popular frameworks for efficient AI and ML development. Combining ExecuTorch with the ubiquitous Arm compute platform, we are accelerating the democratization of AI through new quantized models that empower developers to deploy their applications quicker across more devices, and bring more generative AI experiences to the edge.
Any re-use permitted for informational and non-commercial or personal use only.
Media Contacts
Latest on X
Game-changing quantum breakthrough: Equal1 runs Arm Cortex at 3.3 Kelvin (-269.85°C)!
Equal1's innovative approach could make quantum computing as accessible as traditional servers.
From drug discovery to AI, the future is getting cooler ❄️🌡️ https://newsroom.arm.com/blog/pushing-quantum-computing-boundaries-with-arm-equal1?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk13_tech-strat_na
Here at Arm, we know that prioritizing your wellbeing is key to unlocking your potential.
👉 That’s why we’re committed to nurturing our team’s passions to drive innovation. Scroll through to hear what some of our people have to say about how Arm helps them thrive.
When software architecture isn't standardized, hardware changes can cause issues.
@SchneiderUKI collaborated with us to solve this challenge by bringing in Arm SystemReady, where new hardware works with the software:
SystemReady Implementation at Schneider Electric: Accelerating Development and Enhancing Platform...
How Schneider Electric leveraged Arm SystemReady to accelerate development, enhance platform flexibility and overcome supp...
newsroom.arm.com
Game-changing quantum breakthrough: Equal1 runs Arm Cortex at 3.3 Kelvin (-269.85°C)!
Equal1's innovative approach could make quantum computing as accessible as traditional servers. From drug discovery to AI, the future is getting cooler 🌡️❄️ https://newsroom.arm.com/blog/pushing-quantum-computing-boundaries-with-arm-equal1?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk13_tech-strat_na
AI is causing a fundamental shift in data center infrastructure.
With the flexibility of the Arm compute platform and our relentless focus on energy efficiency and customization, we're at the center of this transformation. Here's why 👇
Arm in the Data Center: The Dawn of a New Era - Creative Strategies
Introduction: 2024 - The Tipping Point for Arm's Data Center Momentum As we close out 2024, the semiconductor indu...
creativestrategies.com
The combined power and performance of Arm and @GitHub is optimizing the developer experience and unlocking advances in AI.
Here's to many more years of partnership and building the future of AI #onArm. 💪
From versatility and cost-effectiveness to scalability and power efficiency, here's 5 key reasons why CPUs are the top choice for AI ⏬
Top 5 Reasons why CPU is the Best Processor for AI Inference - Embedded
Advanced artificial intelligence (AI), like generative AI, is enhancing all our smart devices. However, a common misco...
www.embedded.com
From on-CPU AI processing to chiplet-enabled designs, hyperscalers get more control, efficiency, and power to meet the demands of AI when building #onArm.
@TheNextPlatform shares how custom Arm CPUs are transforming the data center.
Custom Arm CPUs Drive Many Different AI Approaches In The Datacenter
Sponsored Feature Arm is starting to fulfill its promise of transforming the nature of compute in the datacenter, ...
www.nextplatform.com
Arm founding CEO & former chairman, Sir Robin Saxby, has been inducted into the City of London Engineering Hall of Fame!
This acknowledges the transformational impact of Saxby's work on the Arm compute platform, laying the foundation for who we are today: https://newsroom.arm.com/blog/sir-robin-saxby-city-of-london-engineering-hall-of-fame?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk29_exec-comms_na
We're entering a new partnership with @Panasonic Automotive Systems.
To accelerate development cycles in the automotive industry, we'll adopt and extend VirtIO, the device virtualization framework that will help standardize vehicle architecture for SDVs: https://newsroom.arm.com/news/arm-and-panasonic-partner-to-standardize-sdvs?utm_source=twitter&utm_medium=social-organic&utm_content=newsletter&utm_campaign=mk12_automotive_na
We're delighted to welcome Charlotte Eaton back to Arm. 🎉
Charlotte rejoins us as Chief People Officer, drawing on her wealth of experience at Arm and in other leadership roles to empower our people to do the best work of their careers: https://newsroom.arm.com/news/arm-announces-appointment-of-charlotte-eaton-as-chief-people-officer?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk29_exec-comms_na
Demand for the Arm compute platform remains strong in the age of AI.
In Q2 FYE25, we achieved record royalty revenue for Q2, with strong YoY growth thanks to the increased demand for energy-efficient compute and more complex AI chips built #onArm:
Frederique Olivier relies on Arm to capture the beauty of the natural world. 🌎
With the help of Arm, she can shoot images with better quality and source power in extreme conditions.
Discover the world from Frederique's eyes, made possible #onArm: https://newsroom.arm.com/blog/purpose-on-arm-frederique-olivier-photographer?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk08_brand_na
🗓️ Join us on Wednesday November 6 for our next financial results conference call.
After market close, we'll report our earnings for the second quarter of fiscal year 2025, followed by a conference call to review our results and business outlook: https://newsroom.arm.com/news/arm-announces-earnings-release-date-for-second-quarter-fiscal-year-ended-2025?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk29_exec-comms_na
6G is on the horizon, promising ultra-fast speeds, low latency, and seamless digital and physical experiences.
Learn what it takes to bring new AI capabilities to life on 6G: https://newsroom.arm.com/blog/arm-and-alphawave-semi-unite-on-6g-chiplets?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk03_infrastructure_na
Experience new levels of performance and efficiency with the general availability of @GoogleCloud C4A VMs!
C4A features the Arm Neoverse V2-powered Google Axion CPU to support the most demanding AI & general purpose workloads: https://cloud.google.com/blog/products/compute/try-c4a-the-first-google-axion-processor
Congrats @GoogleCloud! 👏
Catch the full conversation to hear why suggesting AI is overhyped "couldn't be further from the truth": https://www.bloomberg.com/news/videos/2024-10-22/arm-ceo-dismisses-ai-bubble-fears-video
#BloombergTech
"AI is going to run on Arm. Period."
Arm is the only company on the planet that can run workloads of any size. There's plenty of opportunity to grow and drive innovation in the space, and we're leading this.
Thanks to #BloombergTech for having us!
At #GitHubUniverse or watching the live stream?
➡️Learn how to maximize performance with GitHub Copilot and GitHub Actions. Catch the team on the Main Stage at 3:45pm PT!
➡️Attending in person? Join our AI roundtable later today - spaces are limited! https://www.arm.com/resources/registration/github-universe-roundtable
Get ready to streamline development with the Arm extension for @GitHub Copilot!
Coming 🔜, the extension will empower developers to build and test software quickly and efficiently on the Arm compute platform, supporting seamless cloud-to-edge deployment: https://newsroom.arm.com/news/ai-driven-tools-on-github?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk24_developer_na
Ready to future-proof your workloads?
Boost performance and cut costs by migrating to Arm. 💪
Companies like @HoneycombIo and @FusionAuth saw major gains with AWS Graviton and so can you. Explore how and access our migration guide to get started: https://newsroom.arm.com/blog/why-developers-are-migrating-to-arm?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk24_developer_cloud-dev
Panasonic Automotive Systems joined us to discuss how AI is revolutionizing the in-vehicle experience in our latest Viewpoints podcast episode.
Discover how AI is shaping the future of driving, making it safer, more intuitive and more comfortable: https://newsroom.arm.com/podcasts/ai-in-the-car?utm_source=twitter&utm_medium=social-organic&utm_content=podcast&utm_campaign=mk12_automotive_na
The Armv9 Cortex-X925 CPU is setting new standards with a 15% IPC improvement!
Get ready for unparalleled performance and efficiency across flagship AI smartphones and PCs. More details: https://newsroom.arm.com/blog/armv9-cortex-x925-cpu-performance?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk04_client_na
For @NVIDIA, transparency is the key to success. 🔑
To build one of the largest companies in the world, NVIDIA CEO Jensen Huang sought to build a porous organization that empowers people to experiment and improve.
Listen to the full Tech Unheard podcast: https://newsroom.arm.com/podcasts/tech-unheard-episode-one-jensen-huang?utm_source=twitter&utm_medium=social-organic&utm_content=podcast&utm_campaign=mk08_brand_na
