RUNNING YOUR RESEARCH SIMULATION IN THE CLOUD?
Cloud computing refers to the on-demand delivery of IT resources via the Internet with pay-as-you-go pricing. Compared to the conventional in-house IT resources, the benefits of cloud computing include no upfront investment, low ongoing cost, flexible capacity, speed and agility, applications not operations and global reach. With the availability of public and private clouds, how do you choose between them?
Public Cloud Computing
Amazon Web Services (AWS) offers the most popular public cloud computing platforms with more than ten major service catalogues, ranging from web service, software service, network service, storage service, database to big data & HPC among others.
High Performance Computing (HPC) on AWS allows one to “buy” special computing resources with high bandwidth, low latency networking, and very high computing capabilities for complex and computing-intensive applications. I decided to try out this service and registered an account in AWS using my credit card number with phone verification giving added security assurance. After that, I simply logged on to the AWS management console and bought the required HPC resources with the online instruction guide and the demonstration video clip posted on YouTube.
As the AWS’ service agreement stated, compute node(s) or server(s) with the required CPU cores, memory, network and operating system (Linux OS was requested as most of HPC clusters are running on Linux) could be provisioned in a few minutes upon submitting my request. However, I have to complete the following tasks in order to integrate the paid compute resources into a ready-to-use HPC cluster, which can then be used to execute a program in parallel manner across all the compute nodes and be shared with my team members.
- Installing compilers and math libraries
- Mounting shared storage to be used as common workspace
- Enabling user account and authentication for my team members
- Hardening the security and firewall to protect the systems
- Integrating the cluster with parallel computing environment
- Installing and configuring job scheduler
The diagram below illustrates the components in the integration a ready-to-use Linux-based HPC cluster. The components shown in green represent the resources can be bought and provisioned by the public cloud service provider such as AWS; while the components shown in grey represent the tasks to be enabled and administered by me. From this perspective, one needs to have sufficient Linux and HPC cluster system administration skills to manage and administrate the above six tasks in order to tap on the public cloud service to feed his/her team’s high performance computing needs.
Besides the cost consideration and ease of access and usage, network bandwidth is another important factor when signing up for the service because the application/program and data to be processed must be uploaded to the servers provisioned in public cloud. In addition the result data will have to be downloaded back as well. Based on my benchmark testing for the network bandwidth from NUS to AWS@SG and AWS@US, the average data transfer speed is in the following scale for your reference.
The above speed was obtained by uploading/downloading a 1GB file. In general, the data transfer speed for AWS@SG is in the scale of 16-20 MB/s; and 340-400 KB/s for AWS@US. The fast data transfer speed between NUS and AWS@SG is expected due to the direct network connection available. The motivation to test the performance between NUS and AWS@US is that more service catalogues, and hardware and OS flavors are provided in the US zone than Singapore’s.
The following online documentation and YouTube video clips are very useful for reference.
Private Cloud Computing in NUS
Computer Centre introduced private HPC cloud services, consisting of Pay-Per-Use service and Condominium service, in early 2012 to enable a more productive use of HPC resources for research. Although built as private cloud and mainly for internal users, the HPC cloud services have almost all the features and advantages offered by the public cloud services. In addition, intensive efforts are made to improve the services with:
• dedicated and ready-to-use HPC cluster for parallel computing,
• minimal provisioning time in one hour,
• ease of sharing among team members,
• large common workspace shared among all clusters,
• single account access and authentication via NUS-ID, and
• professional support for both cluster administration and parallel computing.
To get more details about the HPC cloud services provided by Computer Centre, click here or contact the Research Computing team at hpccloud@nus.edu.sg.
The private HPC cloud services are connected to the campus network, hence the access, authentication and high network bandwidth are well managed and secured.
