KleidiAI Integration Brings AI Performance Uplifts to Google AI Edge’s MediaPipe
At the end of May 2024, we launched Arm Kleidi, which are broad software deliverables and community engagements for accelerating AI across the developer ecosystem. The first part are the Arm Kleidi Libraries for popular AI frameworks, which feature Arm KleidiAI for unleashing CPU performance across AI workloads.
Arm is working directly with leading AI frameworks on KleidiAI integrations, with these already proving to be successful through bringing significant performance improvements to today’s generative AI workloads, including leading large language models (LLMs). For developers, the KleidiAI integrations are completely seamless and transparent, so there is no need to learn additional tools and skills. This allows developers to move faster and extract maximum AI performance for their AI-based applications.
The work has been well-received throughout the industry, notably from the world’s largest technology companies. As part of his COMPUTEX 2024 keynote, Arm CEO Rene Haas shared testimonial videos of executives from Google, Meta and Samsung Mobile talking about how KleidiAI will enable them to accelerate AI innovation across multiple markets.
KleidiAI integration with Google AI Edge
The impact of KleidiAI is becoming a reality, as we have worked with Google AI Edge on its integration into the MediaPipe framework accelerated on the Arm CPU through XNNPACK, which provides support for numerous LLMs including the Gemma 2B LLM. Thanks to the KleidiAI integration, we have seen 30 percent performance improvements in time-to-first token when running our chatbot summarization demo on the Gemma 2B LLM on Samsung’s Galaxy S24 smartphone (Exynos 2400), which is built on Arm CPU technologies.
The performance improvements relate to how many tokens are being processed per second, with the KleidiAI integration enabling around 250 tokens to be processed in one second, making the demo far more responsive. These are exciting results that have positive implications for AI and ML developers, so we are inviting them to test KleidiAI powering Google AI Edge’s MediaPipe using our new Learning Path.
Previously we showed how LLMs can run on the CPU, with Arm being one of the first to demonstrate this. We expanded this work to Google with the chatbot demo shown above, which runs an application that utilizes the MediaPipe APIs and the XNNPACK CPU backend, which is then accelerated by the KleidiAI integration. The great performance shows what is possible for LLMs on the CPU, and how this can enable many real-world AI use cases, including chatbot, smart reply and message summarization.
A series of firsts
This demo is part of a series of firsts for Arm and the wider AI developer ecosystem.
It is the first demonstration of Arm’s partnership with Google AI Edge on MediaPipe and XNNPACK to accelerate AI workloads for developers on the CPU. This is just the start of our work to bring best-in-class performance to XNNPACK, which is an open-source library of highly optimized neural network operators. As XNNPACK has over 7 billion third-party installs, the KleidiAI integration brings the best in AI performance on the CPU to the widest possible market.
“We are excited to support KleidiAI in Google AI Edge’s XNNPACK to accelerate AI workloads on current and future Arm CPUs. This allows AI developers to access existing and new Arm architecture features to deliver outstanding performance that will only improve over time.” Matthias Grundmann, Google AI Edge Lead.
It is also the first in a series of Kleidi integrations that are happening over the coming months, where Arm will be enabling many more LLMs to run as effectively and efficiently as possible on devices on the Arm CPU. By focusing on Kleidi integrations across the software ecosystem, Arm is making AI performance available across the broadest range of hardware and accessible to the broadest community of software developers that are building AI-based applications on Arm, for Arm.
Easy access to AI performance
For developers, the KleidiAI integrations accelerate the development process and unlock AI performance on the pervasive Arm CPU to deliver the very best AI-based experiences on devices. KleidiAI also works across all tiers of CPU that utilize our industry-leading architecture features, like Neon, SVE2 and Scalable Matrix Extension, enabling the development of portable software solutions for application developers.
As Rene Haas said in his COMPUTEX 2024 keynote: “If you don’t have something developers can get access to, the hardware is not going to do you much good.” Stay tuned for more Kleidi integrations and optimizations as we continue to build the future of AI on Arm.
Learn more about the new KleidiAI technical demo here.
Any re-use permitted for informational and non-commercial or personal use only.
Editorial Contact
Latest on X
This is what a day in the life looks like for Graduate Product Manager, Conall Farney. 📹
Since joining us as a Sales Intern, Conall moved into a new role matching his growing passions.
Discover your path with us: https://okt.to/KdWshB
Game-changing quantum breakthrough: Equal1 runs Arm Cortex at 3.3 Kelvin (-269.85°C)!
Equal1's innovative approach could make quantum computing as accessible as traditional servers.
From drug discovery to AI, the future is getting cooler ❄️🌡️ https://newsroom.arm.com/blog/pushing-quantum-computing-boundaries-with-arm-equal1?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk13_tech-strat_na
Here at Arm, we know that prioritizing your wellbeing is key to unlocking your potential.
👉 That’s why we’re committed to nurturing our team’s passions to drive innovation. Scroll through to hear what some of our people have to say about how Arm helps them thrive.
When software architecture isn't standardized, hardware changes can cause issues.
@SchneiderUKI collaborated with us to solve this challenge by bringing in Arm SystemReady, where new hardware works with the software:
SystemReady Implementation at Schneider Electric: Accelerating Development and Enhancing Platform...
How Schneider Electric leveraged Arm SystemReady to accelerate development, enhance platform flexibility and overcome supp...
newsroom.arm.com
Game-changing quantum breakthrough: Equal1 runs Arm Cortex at 3.3 Kelvin (-269.85°C)!
Equal1's innovative approach could make quantum computing as accessible as traditional servers. From drug discovery to AI, the future is getting cooler 🌡️❄️ https://newsroom.arm.com/blog/pushing-quantum-computing-boundaries-with-arm-equal1?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk13_tech-strat_na
AI is causing a fundamental shift in data center infrastructure.
With the flexibility of the Arm compute platform and our relentless focus on energy efficiency and customization, we're at the center of this transformation. Here's why 👇
Arm in the Data Center: The Dawn of a New Era - Creative Strategies
Introduction: 2024 - The Tipping Point for Arm's Data Center Momentum As we close out 2024, the semiconductor indu...
creativestrategies.com
The combined power and performance of Arm and @GitHub is optimizing the developer experience and unlocking advances in AI.
Here's to many more years of partnership and building the future of AI #onArm. 💪
From versatility and cost-effectiveness to scalability and power efficiency, here's 5 key reasons why CPUs are the top choice for AI ⏬
Top 5 Reasons why CPU is the Best Processor for AI Inference - Embedded
Advanced artificial intelligence (AI), like generative AI, is enhancing all our smart devices. However, a common misco...
www.embedded.com
From on-CPU AI processing to chiplet-enabled designs, hyperscalers get more control, efficiency, and power to meet the demands of AI when building #onArm.
@TheNextPlatform shares how custom Arm CPUs are transforming the data center.
Custom Arm CPUs Drive Many Different AI Approaches In The Datacenter
Sponsored Feature Arm is starting to fulfill its promise of transforming the nature of compute in the datacenter, ...
www.nextplatform.com
Arm founding CEO & former chairman, Sir Robin Saxby, has been inducted into the City of London Engineering Hall of Fame!
This acknowledges the transformational impact of Saxby's work on the Arm compute platform, laying the foundation for who we are today: https://newsroom.arm.com/blog/sir-robin-saxby-city-of-london-engineering-hall-of-fame?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk29_exec-comms_na
We're entering a new partnership with @Panasonic Automotive Systems.
To accelerate development cycles in the automotive industry, we'll adopt and extend VirtIO, the device virtualization framework that will help standardize vehicle architecture for SDVs: https://newsroom.arm.com/news/arm-and-panasonic-partner-to-standardize-sdvs?utm_source=twitter&utm_medium=social-organic&utm_content=newsletter&utm_campaign=mk12_automotive_na
We're delighted to welcome Charlotte Eaton back to Arm. 🎉
Charlotte rejoins us as Chief People Officer, drawing on her wealth of experience at Arm and in other leadership roles to empower our people to do the best work of their careers: https://newsroom.arm.com/news/arm-announces-appointment-of-charlotte-eaton-as-chief-people-officer?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk29_exec-comms_na
Demand for the Arm compute platform remains strong in the age of AI.
In Q2 FYE25, we achieved record royalty revenue for Q2, with strong YoY growth thanks to the increased demand for energy-efficient compute and more complex AI chips built #onArm:
Frederique Olivier relies on Arm to capture the beauty of the natural world. 🌎
With the help of Arm, she can shoot images with better quality and source power in extreme conditions.
Discover the world from Frederique's eyes, made possible #onArm: https://newsroom.arm.com/blog/purpose-on-arm-frederique-olivier-photographer?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk08_brand_na
🗓️ Join us on Wednesday November 6 for our next financial results conference call.
After market close, we'll report our earnings for the second quarter of fiscal year 2025, followed by a conference call to review our results and business outlook: https://newsroom.arm.com/news/arm-announces-earnings-release-date-for-second-quarter-fiscal-year-ended-2025?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk29_exec-comms_na
6G is on the horizon, promising ultra-fast speeds, low latency, and seamless digital and physical experiences.
Learn what it takes to bring new AI capabilities to life on 6G: https://newsroom.arm.com/blog/arm-and-alphawave-semi-unite-on-6g-chiplets?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk03_infrastructure_na
Experience new levels of performance and efficiency with the general availability of @GoogleCloud C4A VMs!
C4A features the Arm Neoverse V2-powered Google Axion CPU to support the most demanding AI & general purpose workloads: https://cloud.google.com/blog/products/compute/try-c4a-the-first-google-axion-processor
Congrats @GoogleCloud! 👏
Catch the full conversation to hear why suggesting AI is overhyped "couldn't be further from the truth": https://www.bloomberg.com/news/videos/2024-10-22/arm-ceo-dismisses-ai-bubble-fears-video
#BloombergTech
"AI is going to run on Arm. Period."
Arm is the only company on the planet that can run workloads of any size. There's plenty of opportunity to grow and drive innovation in the space, and we're leading this.
Thanks to #BloombergTech for having us!
At #GitHubUniverse or watching the live stream?
➡️Learn how to maximize performance with GitHub Copilot and GitHub Actions. Catch the team on the Main Stage at 3:45pm PT!
➡️Attending in person? Join our AI roundtable later today - spaces are limited! https://www.arm.com/resources/registration/github-universe-roundtable
Get ready to streamline development with the Arm extension for @GitHub Copilot!
Coming 🔜, the extension will empower developers to build and test software quickly and efficiently on the Arm compute platform, supporting seamless cloud-to-edge deployment: https://newsroom.arm.com/news/ai-driven-tools-on-github?utm_source=twitter&utm_medium=social-organic&utm_content=newsroom&utm_campaign=mk24_developer_na
Ready to future-proof your workloads?
Boost performance and cut costs by migrating to Arm. 💪
Companies like @HoneycombIo and @FusionAuth saw major gains with AWS Graviton and so can you. Explore how and access our migration guide to get started: https://newsroom.arm.com/blog/why-developers-are-migrating-to-arm?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk24_developer_cloud-dev
Panasonic Automotive Systems joined us to discuss how AI is revolutionizing the in-vehicle experience in our latest Viewpoints podcast episode.
Discover how AI is shaping the future of driving, making it safer, more intuitive and more comfortable: https://newsroom.arm.com/podcasts/ai-in-the-car?utm_source=twitter&utm_medium=social-organic&utm_content=podcast&utm_campaign=mk12_automotive_na
The Armv9 Cortex-X925 CPU is setting new standards with a 15% IPC improvement!
Get ready for unparalleled performance and efficiency across flagship AI smartphones and PCs. More details: https://newsroom.arm.com/blog/armv9-cortex-x925-cpu-performance?utm_source=twitter&utm_medium=social-organic&utm_content=blog&utm_campaign=mk04_client_na
