Accelerating Generative AI at the Edge on Arm with ExecuTorch Beta Release
News highlights:
- Combination of Arm Compute Platform with ExecuTorch framework is enabling smaller, optimized models for faster generative AI at the edge
- New quantized Llama models ideal for on-device and edge AI applications on Arm, providing reduced memory footprint and improved accuracy, performance and portability
- 20 million Arm developers can create and deploy more intelligent AI-based applications quicker at scale across billions of edge devices
To realize the true potential of AI, we need to make it accessible to the broadest range of devices and developers. Through collaborating with the PyTorch team at Meta on the new ExecuTorch Beta release, we are fulfilling this mission, bringing AI and machine learning (ML) capabilities to billions of edge devices, as well as millions of developers worldwide.
Generative AI improvements on the Arm Compute Platform with ExecuTorch and new quantized Llama 3.2 1B and Llama 3.2 3B models
The powerful combination of the ubiquitous Arm compute platform, which powers many of the world’s edge devices, with ExecuTorch, a PyTorch-native framework designed to deploy AI models on mobile and edge devices, is enabling smaller, optimized models, including the new quantized Llama 3.2 1B and 3B models. The new quantized models are ideal for generative AI use cases on smaller devices, such as virtual chatbots, text summarization and AI assistants, as they offer a reduced memory footprint, higher levels of accuracy, greater performance and greater portability.
Developers can seamlessly integrate the new quantized models into their applications with no additional modifications or optimizations, saving time and resources. This empowers them to quickly create and deploy more intelligent AI-based applications at scale across a broad range of Arm-powered devices.
As with the new Llama 3.2 large language model (LLM) releases, Arm is optimizing AI performance through the ExecuTorch framework, with this leading to real-world generative AI workloads running faster on edge devices that are built on the Arm Compute Platform. Developers are then able to access these enhancements from day one of the ExecuTorch Beta release.
Accelerating generative AI on mobile with KleidiAI integration
In mobile, Arm’s work with ExecuTorch means virtual chatbots, text generation and summarization and real-time voice and virtual assistants are all running at improved performance entirely on the device on the Arm CPU. This was achieved through integrating KleidiAI, which now introduces micro-kernels optimized for 4-bit quantization, into ExecuTorch via XNNPACK to seamlessly speed-up the execution of AI workloads when running LLMs with 4-bit quantization on Arm Compute Platform. For example, the execution of the prefill stage of the quantized Llama 3.2 1B model will now run 20 percent faster with the KleidiAI integration, leading to speeds of over 400 tokens per second for text generation on some Arm-based mobile devices. This means the end-user will benefit from quicker, more responsive AI-based experiences on their mobile devices.
Learn more about Arm support for ExecuTorch in mobile markets in this blog.
Accelerating real-time processing for edge AI applications in IoT
Meanwhile, in IoT markets, the ExecuTorch work will improve real-time processing for edge AI applications across a broad range of IoT devices, from smart home appliances and wearables to autonomous systems used in retail in industrial IoT. Being able to process more real-time AI tasks on the edge means IoT devices and applications can respond to their environments in milliseconds, which is crucial for safety and functionality.
ExecuTorch can be leveraged across Arm’s Cortex-A CPUs and Ethos-U NPUs to accelerate the development and deployment of edge AI applications. In fact, through combining ExecuTorch with the Arm Corstone-320 reference platform (which is also available as an emulated Fixed Virtual Platform), the Arm Ethos-U85 NPU driver and compiler support into one package, developers can start creating their edge AI applications months before platforms arrive on the market.
Learn more about Arm support for ExecuTorch in IoT markets in this blog.
More accessible, faster edge AI experiences
We believe that ExecuTorch has the potential to be one of the world’s most popular frameworks for efficient AI and ML development. Combining ExecuTorch with the ubiquitous Arm compute platform, we are accelerating the democratization of AI through new quantized models that empower developers to deploy their applications quicker across more devices, and bring more generative AI experiences to the edge.
Any re-use permitted for informational and non-commercial or personal use only.
Media Contacts
Latest on X
Thank you, #GITEXGlobal, for including us to speak on this moment in technology where we can truly make a difference. Tamika Curry Smith was on the ground to share our commitments around #DEI and #AI.
The future is bright and built #onArm.
It's almost time! ⏰
In just a few hours, Arm CEO Rene Haas joins #BloombergTech to discuss everything from Arm's story so far to the opportunity in AI and what's next for the semiconductor industry.
👇 Watch the stream at 2:35 pm BST on @BloombergLive: https://x.com/i/broadcasts/1OyKAZovbvLGb
🏆 We've won the 2024 @Reuters Automotive D.R.I.V.E Honours Award for Innovation!
This award recognizes our commitment to advancing automotive design and production with our automotive technologies and CSS roadmap, launched back in March: https://1.reutersevents.com/LP=37549 #DRIVEHonours
We're transforming the automotive industry.
Our compute platform is key to developing SDVs with enhanced capabilities and safety features that will help enhance driving experiences. 🚗
Discover more on our blog ➡️: https://newsroom.arm.com/blog/software-defined-vehicles-future-automotive-industry?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk12_automotive_na
Thanks for having us, #OCPSummit24!
With demos, daily speaker sessions, & new chiplet announcements with Arm Neoverse CSS, we showed how to tackle the scalability, design & development challenges of AI on the Arm compute platform.
Catch our highlights: https://newsroom.arm.com/blog/key-takeaways-from-ocp-global-summit-2024?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk03_infrastructure_na https://x.com/i/web/status/1847413131980788038
Join Rene Haas next week at #BloombergTech to hear how we're building the future of computing #onArm.
🔴🎥 Catch Rene in person on October 22 or tune in to the live stream via @BloombergLive at 2:35 pm BST. https://events.bloomberglive.com/bloomberg-technology-summit-london-2024/agenda/speakers/3407898
✅ Seamless compute scalability
✅ Trusted safety and security
✅ Accelerated software readiness
The Arm compute platform places us at the forefront of technology to deliver scalable, efficient, and innovative automotive solutions.
More in our blog 👉: https://newsroom.arm.com/blog/accelerating-adas-adoption?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk12_automotive_na
⏰ There are only a few hours left of #OCPSummit24!
This is your final chance to head to the Arm booth at #B12 to see chiplets for AI, powered by Arm Neoverse CSS, and the Arm-based tech enabling the sustainable AI data center.
🏃🏃🏃
What makes a company truly resilient and strong?
@NVIDIA CEO Jensen Huang reflects on the moments that helped shape him as a leader, and his advice to others for persevering when times get tough.
Catch the full episode on the Tech Unheard podcast: https://newsroom.arm.com/podcasts/tech-unheard-episode-one-jensen-huang?utm_source=twitter&utm_medium=social-organic&utm_content=podcast&utm_campaign=mk08_brand_na
"The insatiable demand for AI performance is putting immense pressure on today's datacenters."
At #OCPSummit24, Eddie Ramirez shared details on the importance of chiplet technology and ecosystem collaboration on the Arm compute platform to scale AI innovation.
Stay tuned for… https://x.com/i/web/status/1846930772776964347
We're at #OCPSummit24! 👋
If you're here, swing by and chat with us, or head to one of our talks to learn about the exciting possibilities of custom chiplets with Neoverse CSS and the Arm Total Design ecosystem.
📢 The general availability of Arm Neoverse CSS-based Microsoft @Azure Cobalt 100 VMs has just been announced!
This brings the performance, efficiency and flexibility of Neoverse CSS-based instances to millions of users for the age of AI: https://newsroom.arm.com/blog/arm-powered-microsoft-azure-cobalt-100-vms?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk03_infrastructure_na
🚀 @Vivo_GLOBAL's X200 series has launched!
Will Abbey, EVP and Chief Commercial Officer, joined vivo on stage to celebrate the performance and AI capabilities of the new X200 series built #onArm with @MediaTek's Dimensity 9400 chipset.
@RebellionsAI @alphawavesemi @Cadence @ProteanTecs Discover more about Arm Total Design and its role in the future of AI-powered computing at #OCPSummit24, where you can join us for talks, demos, and partner showcases to learn about the next era of built-for-AI silicon - #onArm.
@RebellionsAI @alphawavesemi 3️⃣ Arm Total Design partners including @alphawavesemi, @Cadence and @ProteanTecs are validating their third-party IP products with CSS on advanced nodes - helping to provide an out-of-the-box software experience with Neoverse CSS-based custom silicon on leading edge nodes.
@RebellionsAI 2️⃣ Neoverse CSS and Arm Total Design are helping to create the hardware foundation for the sustainable AI datacenter with 2x new chiplets announced from Alcor Micro and @alphawavesemi, both on Arm CSS. 💪
1️⃣ Together with Samsung Foundry, ADTechnology and @RebellionsAI, we're bringing an AI CPU chiplet platform to market, based on Arm Neoverse CSS V3.
This promises to deliver unparalleled performance and optimal power efficiency for cloud, HPC and AI/ML training and inference.
🆕 After doubling in size in just a year, we're excited to share new updates on the chiplet innovation we're seeing from the Arm Total Design ecosystem. 🧵
By uniting on Arm CSS, we're building solutions for the AI datacenters of the future. Details: https://newsroom.arm.com/news/arm-total-design-sustainable-ai-datacenter?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk03_infrastructure_na
Discover why all infrastructure market leaders choose Arm Neoverse to power their data center transformation at #OCPSummit24.
📍 Booth #B12
👉 Catch one of our talks, agenda below
📅 Tuesday-Thursday
See you there!
Arm is ✨the✨ compute platform for AI
From cloud to edge, our CPUs help software developers & our partners create highly customized silicon solutions.
➕They’re the ideal companion if you’re looking for a flexible heterogeneous computing approach to AI: https://newsroom.arm.com/blog/arm-compute-platform-ai-accelerator-technologies?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk13_tech-strat_na
We've committed to join @imec_int's Automotive Chiplet Program! 🤝
Announced today, the program brings together industry leaders to evaluate, shape and define the best chiplet architecture for future vehicles, prioritizing performance and safety.
Automotive Chiplet Program: first partners | imec
Arm, ASE, BMW Group, Bosch, Cadence, Siemens, SiliconAuto, Synopsys, Tenstorrent and Valeo commit to join imec’s Automotive Chiplet Program.
www.imec-int.com
.@MediaTek's Dimensity 9400 Chipset is here! 🚀
This flagship SoC features Armv9.2 CPU architecture and Immortalis-G925 GPU to to put PC level performance in the palm of your hands.
This enables a new world of immersive gaming and edge AI applications:
Arm and MediaTek Deliver Performance and Efficiency for AI Experiences
From smartphones to IoT, MediaTek leverages the Arm cutting-edge architecture to bring new possibilities to the market.
www.arm.com
"We're at the beginning of a new industrial revolution"
Rene Haas is joined by @NVIDIA CEO Jensen Huang on the 1st episode of Tech Unheard.
🎧 Listen in as Jensen introduces the AI factory, where intelligence is now manufactured at large volumes: https://newsroom.arm.com/podcasts/tech-unheard-episode-one-jensen-huang?utm_source=twitter&utm_medium=social-organic&utm_content=podcast&utm_campaign=mk29_exec-comms_na
🤝 Together with @AIatMeta, we're driving open innovation to optimize LLM performance.
Hear from Sy Choudhury, Director of AI Partnerships, as he shares insights on our long-standing collaboration to accelerate AI software development and empower the developer community.
