The slow demise of an industry

It is a sad week in the High Performance Computing (HPC) industry as one of the best known giants in the field is acquired. Hewlett Packard Enterprises (HPE) is in the process of closing on a 1.3 billion dollar deal to purchase supercomputer maker Cray. This is the second supercomputing company to be purchased by HPE within 3-years. HPE purchased Silicon Graphics Incorporated in 2017.
This acquisition brings the global total of true supercomputing manufacturers selling product in the U.S. down to three. There are other companies offering HPC platforms, but only HPE, IBM and Atos remain as manufacturers of truly integrated HPC platforms.
I know this raises a question, what is an integrated HPC platform? I will first start by telling you a little history of HPC. In 1995, the biggest computers were very expensive mainframe systems manufactured by only a couple of companies. They could do the equivalent work of 10-12 normal computers, but cost two to three hundred times more. Out of the need for more powerful computing systems at a lower cost, a group of engineers developed a programming method that allowed computers to pass messages between one another. The Message Passing Interface (MPI) was born. I was among the team members that developed the standards behind that interface.
MPI brought about a whole new realm of computing called distributed computing. It is the backbone of almost every HPC system, including some aspects of modern cloud computing. MPI allowed smaller research organizations with a low budget to take even their used desktop computer systems that were being pulled out of service and link them together to create massive computing platforms. They were termed Beowulf clusters, named after the first one created by Thomas Sterling and Donald Becker at NASA in 1994.
This research sparked a brand new industry and a few short years ago there were hundreds of HPC manufacturing companies. Over the years a few big users of HPC felt the need for more robust systems than the hodge-podge, stick-it-together-with-duct-tape-and-network-switches type systems. This birthed companies like Cray and SGI. IBM and Atos/Bull had been around manufacturing the expensive mainframes before entering the HPC market.
The things that set the likes of HPE, IBM, SGI, Cray and Atos apart is the manufacturing of these Beowulf style clusters into integrated racks including customized computer motherboards, cooling methods, and interconnects in order to shrink the size and power utilization, effectively reducing the total cost of ownership of the building-sized computing platforms. The other manufacturers stuck to racking commodity server hardware and manually cabling the networks and infrastructure. The initial cost of these systems is lower, but they cost more to maintain over time.
It saddens me to see us down to three companies working on the bleeding edge of HPC hardware because, much like any other industry, as the competition dies off or merges, the prices increase and the innovation decreases. I miss the days when every researcher in computer science was building their own HPC systems and we all worked together to solve the problems and innovate new solutions. Quantum Computing will probably save the day with innovation, but that’s a topic for another day.