DPUs: Dedicated Data Chips Tackle Tedious Tasks

Data Processing Units (DPUs), also known as Infrastructure Processing Units (IPUs), are made to handle the behind-the-scenes work of cloud computing: exacting and time-sensitive security tasks like packet inspection and hypervisor management so video streams can flow and artificial intelligence (AI) algorithms can unearth unexpected trends.

And that’s exactly why specialized processors are rapidly becoming an exciting segment in the industry. Network, storage, and security tasks can now consume as much as 30 percent of a server’s CPU cycles for the resources required by an instance.

A dedicated processor optimized for these tasks–one that is better than them, in fact, than a central processing unit (CPU)–creates an opportunity for customers to better focus their CPU cycles towards paying workloads instead of infrastructure tasks, yielding TCO benefits in power and capex.

Some notable announcements include:

• In July, Marvell Technologies announced its OCTEON 10 family of DPUs. Built with Arm Neoverse N2 and TSMC’s 5 nanometer process, OCTEON 10 promises to deliver 3x more performance than its predecessor and 4x the performance per watt.
• Intel recently announced its IPU code-named Mount Evans with a leading US based cloud service provider that will combine 16 Arm Neoverse N1 processors and other accelerators into a single chip. (Update 27/10: Intel announced that Google is its cloud partner.) Intel has also developed IPUs based around FPGAs.
• NVIDIA, meanwhile, announced that its Bluefield-3 DPU, containing its Mellanox technology and 16 Arm Cortex-A78 processors, will be able to replace 300 traditional processor cores. Samples are expected in the first quarter of 2022. Later, Bluefield-4 will double bandwidth to 800Gbps.
• VMWare, Dell, Pensando Systems, Palo Alto Networks, Canonical, Supermicro, NetApp and others have launched products and/or initiatives to take more advantage of DPUs.

DPUs and IPUs are becoming the control point of the cloud network and that is enabling some profound improvements in security and composability.  From a security standpoint, these technologies are the gold standard in system security.

Whether you are talking about booting secure and attested software, threat detection or deep packet inspection of all incoming network traffic, none of that is practical at cloud scale without DPUs.

DPUs: A Decade of Innovation

Efforts to develop DPU-like systems to handle network, security and storage functions began around a decade ago. In 2014, Amazon Web Services, for instance, began to work with Annapurna Labs, a startup designing Arm-based chips, to develop a system for accelerating storage tasks. It bought Annapurna the following year and in 2017 debuted the Nitro System for managing a broad menu of tasks like hypervisor management, virtual private clouds, elastic block storage and high-performance networking. (The Nitro System is comprised of three main parts: the Nitro Cards, the Nitro Security Chip, and the Nitro Hypervisor.)

Since then, Nitro has become part of the foundation of modern EC2 instances, AWS Outpost, Mac Instances and other services.

Not only has the Nitro System improved customer experiences, Amazon CTO Dr. Werner Vogels says it gives AWS more bandwidth to innovate. Since launching Nitro, the number of instances available on AWS has grown by 4x.

“Whether you chose to look at something as a problem or an opportunity can have large effect on how you deal with it. We have chosen to look at the limitations of the traditional hypervisor as an opportunity to create a completely new architecture,” he writes in his All Things Distributed blog. “Now with the Nitro System, our customers enjoy better performance, enhanced security, and a broader set of instance types to choose from. And we’re not done yet.” 

Anatomy of a Data Processing Unit (DPU)

DPUs are an evolution of “SmartNICs” like AWS Nitro and FPGAs, which were integrated into cloud operations for offloading administrative tasks. DPUs aren’t offload devices. They are dedicated, co-equal processors that boot their own operating system (OS) independent of the host OS and manage their own memory and storage.

They also add programmable packet processing (with P4, for example) to a base of secure boot, high-speed networking and performant, low-power general purpose cores. Not only does carving out these tasks into a separate domain improve performance, it expands what cloud providers can accomplish with them. If the CPU is the CEO, the DPU is the COO.

While hyperscale cloud providers are currently the main consumers of these products, the technology will also spread to enterprise data centers and high-performance computing laboratories. . With Project Monterey, VMWare is moving the ESXi hypervisor onto DPUs and recognizing the benefits of specialized hardware to deploy enhanced virtualized services.

This will enable faster adoption and cloud-native software in enterprise and telco markets. Project Cassini and Arm SystemReady, meanwhile, give developers, hardware manufacturers and their customers a glide path toward interoperability across the stack.

The Next Frontier

The technology will likely percolate to other markets and use cases as well. The AI and machine learning (ML) and security requirements of new applications—think high-end robotics or autonomous driving— could well prompt designers to adopt specialized data processing into their product designs.

Device architecture will also become increasingly complex. NVIDIA’s Bluefield-4, for instance, will contain multiple CPUs and GPUs to achieve a quadrillion operations per second. At some point, neural processing units (NPUs) could become part of DPUs.

This is a space well worth watching.