AI/ML Drives Greater Enterprise HPC Adoption
By Tan Chee Chiang & Kuang Hao, Research Computing, NUS Information Technology
Recent trends indicate that the proliferation of AI/ML applications has a positive effect on the adoption of HPC technologies by enterprises. We will analyse why that is happening and how it will benefit the HPC practitioners. This synergistic development of AI and HPC has also led to the renaming of this periodical to HPC-AI Newsletter.
Why HPC resource/technology sharing between research and enterprise didn’t work previously?
In the 1990s, when the research/HPC community started embracing the x86 processor and the business community started considering Linux OS for enterprise systems, we attempted without success to bring HPC and enterprise system users together to share resources. We met with little success as there was no common problem or application between the two communities to work on.
In the 2000s, when Big Data analytics became a big thing (pardon the pun) for both research and enterprise, we started to have some successes in resource/technology sharing, even though it was limited to mainly storage systems. Even though businesses were embracing data analytics, it was generally at a relatively small scale and hence HPC wasn’t required.
What make it work now?
It was not until the 2010s, that we started to see greater adoption of HPC by enterprises due to the renaissance of AI, particularly in machine learning (ML) and deep learning (DL). ML/DL requires HPC technologies and resources to work due to its high computation and data processing demands.
The HPC community has been dealing with such challenges for a long time in the research computing landscape. In the past 10 years, we have also seen more interaction between the HPC and AI community. Here is an analysis we did to validate that trend.
Over the last 12 months, the top-5 required software/programming languages indicated in new HPC account application at NUS IT are Data Analytics or AI–related, as shown in the chart below.
We crawled through 2,976 articles from a HPC magazine over 12 months and found that 57.2% of them mentioned an AI term at least once. In the chart below, we have captured the frequency of such AI terms that were found in the HPCwire. It’s not surprising to find Machine/Deep Learning at the top of the ranking since HPC is required for such compute-intensive processing.
With enterprise computing and HPC having a common popular application now, enterprise adoption of HPC is expected to be a natural progression.
The above development has also prompted us to rename this periodical from “HPC” to “HPC-AI” Newsletter.
Why does it matter to HPC practitioners?
With increasing enterprise adoption of HPC due to AI/ML implementations, one obvious benefit will be greater career opportunities. When large enterprises implement AI/ML, besides recruiting Data Scientist, they may need HPC practitioners also if they are operating their systems in-house.
For HPC practitioners to be able to add value to enterprise AI/ML projects, they will have to pick up production application deployment skillsets that are usually not required in research computing support. Such skillsets include but are not limited to web application deployment, application HA implementation, real-time data streaming and transaction, and cloud deployment. Going forward, we hope to see a new generation of HPC practitioners that will help to drive the further development of AI/ML in both research and enterprise.
