HPC in the Cloud? Yes, No and In Between

Is 2021 the watershed moment for HPC in the cloud? The question has been asked of the coming year repeatedly for the last decade and the response usually has been an unqualified yes, well, sure, probably.

The major cloud vendors have all rolled out HPC cloud services and steadily expanded them. AWS, for instance, recently unveiled the AWS Graviton 2-based C6gn instance that can deliver up to 100 Gpbs networking while providing 40% better price performance over contemporary processors. This is a key enabler for HPC applications which have historically gone widely parallel to obtain the desired performance.

RIKEN, the research institute behind Fugaku, the reigning system on the Top 500 list two times in a row, recently announced it would offer cloud services. Perhaps leading in an innovative way, this public/private partnership between the worlds #1 system and a growing trend for HPC in the cloud is one to watch.

HPC users, meanwhile, have responded by porting increasingly complex workloads like EDA and drug design to these services and for good reason. Universities, research organizations and even traditional commercial HPC users like oil & gas face increasingly tight budgets. Moving to the cloud gives them the freedom to scale up (or ramp down) capacity.

Projects can run in parallel rather than wait in a queue. And it works on Arm in a big way: benchmarks on the 80-core Altra, an Arm-based processor from Ampere Computing, running in the cloud show a time reduction of 10-42% on various applications. Time, money and headaches will all be reduced.

HPC enables access to a wider portfolio of tech

HPC in the cloud will also potentially give organizations access to wider portfolio of technology. We’re entering an era where gains in performance will be achieved through processor innovations and system design more than Moore’s Law.

If you look at the most recent Top 500 list you will find five different processor architectures and four different interconnect technologies are represented in the top ten entries.

Fugaku, for example, is built around the Arm-based Fujitsu A64FX processor and Fujitsu’s Tofu Interconnect D: other organizations are already examining ways to incorporate the technologies into their own projects. As these different technologies show advantages for certain workloads, organizations can direct their attention to the most efficient/effective offerings.

And perhaps most importantly of all, HPC in the cloud will also dramatically expand the number companies that can take advantage of broader capabilities offered in the range of petascale today and perhaps growing to exascale in the future. The affordability of shared use of workload specific hardware at scale has the potential to help us solve the monumental tasks facing humanity.

Consider genomics. DNA data is doubling every seven months and will likely outstrip the growth of other big data generators like YouTube. The information it contains will hold the key to improving crop yields and inventing new vaccines. But without high-performance systems that can be spun up and down quickly, a gap will exist between the knowledge base and the base of scientists that need it.

Slow progress in High Performance Computing

So – plenty of potential here. But why such slow progress? Why is it that HPC is later to the cloud party than most other markets?

One reason is the software. The cloud native juggernaut has yet to pervade HPC because many applications are still written in Fortran, a reflection on the complexity of moving to new environments. Another reason is the environment: cloud systems provide environments with virtualization, packaging and control planes that are different from those familiar to HPC users. There is a noticeable gap here – one that is begging for a solution. (Spoiler alert: watch for news on this front soon). All said, progress is being made: take a look at the cloud-based seismic mapping tools from S-Cube from Imperial College, London. Tools and languages are following as well.

Then there is the psychological barrier. The not-invented-here barrier runs deep in HPC and for good reason. For decades, the HPC community pioneered ideas that have been picked up by the hyper-scalers. For that trend to turn around takes collaboration and communications. It also means that we need to surrender some control of our system architecture. National security considerations—part of HPC since the days of Colossus and ENIAC—will be another factor.

A hybrid environment

But perhaps most important of all is the fear of omission: will a shift to the cloud close off avenues of inquiry that could lead to fundamental breakthroughs? HPC requires the risk-taking, long-term R&D often best conducted inside national laboratories and other independent organizations. I personally don’t think this kind of research will get marginalized by the growth of HPC in the cloud, but it’s a concern that needs attention. Clearly it would be good news to see more public-private partnerships here as we appear to be witnessing with RIKEN.

In conclusion, I think we are seeing a movement toward a hybrid environment, led by commercial HPC users migrating to the cloud. Academia will follow while government agencies will keep a foot firmly in both camps and slower to adopt. Ideally, partnerships in business and a healthy mix of users will encourage further growth and provide a pathway for reaping the economic and technical benefits without losing the Manhattan Project heft that advancing the state of the art in HPC will require. 

If anything, it will make the next five years quite interesting.

This blog originally appeared on HPCWire.