Accelerating and Scaling AI Inference Everywhere with New Llama 3.2 LLMs on Arm
- Significant performance improvements from cloud to edge when running Meta’s new Llama 3.2 release on Arm CPUs, enabling future AI workloads
- Collaboration between Meta and Arm enables faster innovation in use cases such as personalized on-device recommendations or automating routine tasks
- A decade of AI investment from Arm and extensive open-source collaboration enables new LLMs ranging from 1B to 90B seamlessly on Arm compute platforms
The ongoing rapid evolution of AI means new versions of large language models (LLMs) are emerging on a regular cadence. While LLMs running everywhere, from cloud to edge, are needed to realize the full potential and opportunities of AI, these are driving significant compute and energy demands. The ecosystem is rallying to find solutions to this challenge, and releasing new, more efficient open-source LLMs to enable a broad range of AI inference workloads at scale and bring new, accelerated AI experiences to the user, quicker.
Through our collaboration with Meta enabling the new Llama 3.2 LLMs on Arm CPUs, we are demonstrating the powerful combination of open-source innovation and the Arm compute platform in tackling this challenge. Arm’s ongoing investments and work with new LLMs like this one mean the ecosystem can automatically see the benefits of running AI on the Arm CPU, making it the platform of choice for developers targeting their AI inference workloads.
Accelerated cloud to edge AI performance
The availability of smaller LLMs that enable fundamental text-based generative AI workloads, such as Llama 3.2 1B and 3B, are critical to enabling AI inference at scale. Running the new Llama 3.2 3B LLM on Arm-powered mobile devices through the Arm CPU optimized kernel leads to a 5x improvement in prompt processing and 3x improvement in token generation, achieving 19.92 tokens per second in the generation phase. This means less latency when processing AI workloads on the device and a far faster overall user experience. Also, the more AI processed at the edge, the more power that is saved from data traveling to and from the cloud, leading to energy and cost savings.
Alongside running small models at the edge, we are also able to run larger models, such as Llama 3.2 11B and 90B, in the cloud. The 11B and 90B models are a great fit for CPU based inference workloads in the cloud that generate text and image, as our data on Arm Neoverse V2 shows. When we run the 11B image and text model on the Arm-based AWS Graviton4, we can achieve 29.3 tokens per second in the generation phase. When you consider that the human reading speed is around 5 tokens per second, it’s far outpacing that.
AI will scale quickly with open-source innovation and ecosystem collaboration
Having new LLMs like Llama 3.2 openly available is vital. We see that open-source innovation moves incredibly fast: In previous releases the open-source community has had new LLMs up and running on Arm in less than 24 hours.
We are further bolstering the software community through Arm Kleidi, where we are working to enable the entire AI technology stack to leverage this optimized CPU performance. Kleidi unlocks the AI capabilities and performance of Arm Cortex and Neoverse CPUs on any AI framework with no application developer integration required.
Through our recent Kleidi integration with PyTorch and the in-progress integration with ExecuTorch, we are enabling seamless AI performance benefits for developers on the Arm CPU from cloud to edge. The Kleidi integration with PyTorch leads to a 2.5x faster time-to-first token on Arm-based AWS Graviton processors when running the Llama 3 LLM.
Meanwhile at the edge, the KleidiAI libraries are accelerating the time-to-first token for Llama 3 using llama.cpp by 190 percent on the Arm Cortex-X925 CPU compared with the reference implementation.
Building the future of AI
When you combine the flexibility, pervasiveness and AI capabilities of the Arm compute platform with the expertise of industry leaders like Meta, you unlock new opportunities for AI at scale. Whether it’s on-device LLM performing tasks on your behalf by understanding your location, schedule, and preferences, or enterprise use cases to make us more productive and focus on the higher value tasks at work, the integration of Arm’s technology is paving the way for a future where devices are not just command and control tools but proactive assistants that enhance the user’s overall experience.
The AI performance uplifts on the Arm CPU through the new Llama 3.2 LLM are impressive, and we believe that these open collaborations are the best way to enable AI innovation everywhere, in the most sustainable way possible. Through new LLMs, the open-source community and Arm’s computing platform, we are building the future of AI, with more than 100 billion Arm-based devices set to be AI ready by 2025.
Additional resources
For mobile and edge ecosystem developers, Llama 3.2 runs efficiently across Arm Cortex CPU based devices. See our documentation for developer resources.
Developers can access Arm from all the major cloud service providers running Llama 3.2 in the cloud on Arm Neoverse CPU. See our documentation for getting started.
Any re-use permitted for informational and non-commercial or personal use only.
Media Contacts
Latest on Twitter
🐻 We sponsored the AI for Bears challenge with @fruitpunchai.
Over 10 weeks, 19 participants learned about human-wildlife conflict, AI development flows, and teamwork.
See how they used this knowledge to tackle bear conservation issues 👉 https://community.arm.com/arm-community-blogs/b/ai-and-ml-blog/posts/ai-for-bears-challenge?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk02_iot_na
The future of IoT is built #onArm.
To discover how the Arm ecosystem delivers efficient performance in edge AI devices, tune into @pnwilliamson's keynote at @EdgeImpulse's #Imagine2024 event.
🗓️: September 24, 2024
📍: Mountain View, CA or Virtual
👉: https://edgeimpulse.com/imagine
↔️ Expanded memory capabilities for even more workloads
⚡ 60% better performance than the previous instance
🔋 The highest level of energy efficiency so far
Meet @AWSCloud's new Graviton4-powered, memory-optimized EC2 X8g instances, built #onArm.
Now available: Graviton4-powered memory-optimized Amazon EC2 X8g instances | Amazon Web Services
New memory-optimized X8g instances offer up to 3 TiB DDR5 memory, 192 vCPUs, and 50 Gbps network bandwidth, design...
aws.amazon.com
Harnessing the power of AI is easy when you have the right tools and resources. 💪
Great to share this insight with developers at #PyTorchConf.
$ARM's now a part of the PHLX Semiconductor Sector Index (SOX)!
We're thrilled to join an esteemed list of leading semiconductor companies, reinforcing our position as the foundational compute platform for the age of AI across every technology touchpoint https://newsroom.arm.com/news/arm-joins-semiconductor-sector-index-sox?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk29_exec-comms_na
Experience the world in a whole new way with @Snap's new @Spectacles, built #onArm. 👓
The new AR glasses benefit from our leadership in performance and energy efficiency to unlock new immersive experiences with Lenses. We even got to try them out! https://newsroom.snap.com/sps-2024-spectacles-snapos
💪 Our impact on the world is important to us.
🌍 From reducing our carbon emissions to enabling equitable healthcare in communities, we're all in on making a difference.
See how we're doing so far.
🎙️ Introducing Tech Unheard, a new podcast from Arm CEO, Rene Haas.
Join Rene and other tech leaders from across the industry for unique, behind-the-boardroom-door conversations.
We're kicking things off with @NVIDIA's Jensen Huang on Oct 9: https://newsroom.arm.com/news/arm-and-npm-announce-new-tech-unheard-podcast?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk29_exec-comms_na
Arm makes running MLOps workflows a breeze on @GitHub Actions.🏃
💪 With Arm Runners, developers can benefit from optimized performance, effortlessly turning workflows into powerhouses of efficiency.
@PyTorch 🙋Who's coming? Get more info here:
Schedule | LF Events
18-19 September 2024 | San Francisco, California You must be registered for PyTorch Conference 2024 to participate i...
events.linuxfoundation.org
At #PyTorchConf, join Gian Marco Iodice and Pareena Verma for more details on how to unlock performance gains with @Pytorch and Arm Kleidi. 🚀
Just over a year ago we began our journey as a public company.
⏩ Since, we've moved at pace to continue building the industry's most efficient and performant compute platform for the age of AI.
Here's a look at just some of our highlights along the way:
We're optimizing AI across the ML stack. 💪
That's why we've partnered with @PyTorch and ExecuTorch to integrate Arm Kleidi Libraries directly to each framework.
This gives application developers enhanced AI performance on the CPU without extra work: https://newsroom.arm.com/news/pytorch-kleidi-integrations-cloud-to-edge?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk13_tech-strat_na
Congrats to our Executive Vice President and Chief Legal Officer, Spencer Collins for the recognition as one of the top in-house legal leaders by the @FT.
Spencer’s leadership and expertise remains crucial to the success of the Arm compute platform: https://www.ft.com/content/e9a36690-fa99-4525-bf68-42132c77e553
We recently hosted an event bringing together experts on Edge AI. 🚀
This included our own SVP & GM of IoT @pnwilliamson, who showcased the power of Arm Ethos-U85 running Tiny Llama2.
See it in action.👇
@TinymlF
We're thrilled to welcome Young Sohn to our Board of Directors! 🎉
With diverse experience in the semiconductor industry and a long history of working with Arm, Young's contributions will be crucial to the growth of the Arm compute platform in the AI era: https://newsroom.arm.com/news/appointment-of-young-sohn-to-arm-board-of-directors?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk29_exec-comms_na
We're joining the @PyTorch Foundation as a premier member!
Together we can ensure the integration of PyTorch's framework with Arm's software stack to accelerate AI development on Arm.
This forms part of our continued investment in software & open source: https://pytorch.org/blog/arm-joins-pytorch/
Congrats @Nuro!
We're proud to power the Nuro Driver platform with the server-class performance & safety capabilities of Arm Neoverse V3AE technology.
In this new era, together we're enabling OEMs with the safety-enabled compute performance for autonomous, AI-enabled vehicles.
We're taking gen AI to the next level with 3 🆕 practical demos running entirely on the Armv9 CPU on your smartphone.
1️⃣ Real-time voice assistant
2️⃣ Voice note summarization
3️⃣ Group chat summarization
This just scratches the surface of what's possible: https://okt.to/BeX8UH
Are you ready to build native apps on Windows on Arm?
We've got the tools and resources you need to get started.
This is how you build enhanced user experiences.
Here's to 3+ years of SOAFEE and everything that's to come with http://SOAFEE.next. 🎉
With 120+ members and counting, we're proud to celebrate the collaborative efforts of the entire automotive ecosystem to bring software-defined, AI-enabled vehicles to life:… https://twitter.com/i/web/status/1831330305568874744
Do you build @VulkanAPI apps? We’ve got exciting news for you!
🔜 Native support for Windows on Arm is coming.
🎬 See it in action:
Processing complex AI workloads on the device can be slow. 🦥
Arm KleidiAI is changing this, bringing best performance without additional optimization.
We show what's possible when it's integrated with @Google's MediaPipe & XNNPACK on Android.
Meet the next generation of optimized ML workloads.👇
⚡ 30-50% speedups for various ML libraries
💪 3.1x run time improvement with Photon
💰 Lower cost
Get this and more with Arm Neoverse-based @AWSCloud Graviton on the @databricks ML Runtime cluster.
Unlock Faster Machine Learning with Graviton | Databricks Blog
okt.to
