Creating Your Own Private HPC In The Cloud
Yeo Eng Hee, Research Computing, NUS IT.
Have you ever wished to easily set up your own private HPC server in the cloud to run your computations without getting professional help? And after you are done with your computations, destroy everything so that you do not have to pay for idle cloud resources?
Here in NUS IT’s Research Computing team, we have been exploring and using the cloud for about 5 years, and even though the cloud has many tools to help a tech-savvy person learn and get started, we realise that for many others, there is a relatively steep learning curve.
To complicate matters further, there are now many competing public cloud providers, and each provider has their own tools for managing their cloud resources. This means that for the average researcher, there will be another learning curve when switching to another cloud provider.
Terraform – A Third Party Tool
Therefore, there is a need for a common tool that can be used to manage the different environments from the different cloud providers. Terraform by HashiCorp is such a tool, and it supports not only the major cloud providers (AWS, Azure, Google) but also many others, including Oracle, IBM, Digital Ocean, etc. Terraform is also open-source, and can be downloaded for individual use.
Terraform is rapidly becoming one of the popular tools to manage not just public cloud infrastructure, but also private ones, network appliances, software-as-a-service (SaaS) and platform-as-a-service (PaaS)[i]. Terraform uses an easy-to-read configuration language that is used to declare the cloud resources, very similar to the common JSON format. Therefore, it is easy to learn and modify.
In the subsequent paragraphs, we show how Terraform can be used to easily deploy a complete VPC with a running server instance, which can also be easily destroyed after it is not needed.
A Sample Terraform Configuration
The code below shows how, with a few lines, Terraform can be used to define an EC2 instance in the AWS cloud:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.27"
}
}
required_version = ">= 0.14.9"
}
provider "aws" {
profile = "nus-it-hpc"
region = "ap-southeast-1"
}
resource "aws_instance" "app_server_intel" {
ami = "ami-055d15d9cfddf7bd3"
instance_type = "m6i.metal"
tags = {
Name = "My Intel Server"
}
}
The first section defines the “terraform” parameters, including the “provider” which, in this case, is “aws”. The provider is plugin written by Terraform to interact with the appropriate cloud providers, and is access from the source, which is the Terraform Registry, as “hashicorp/aws”.
The next section defines the parameters needed by the provider, namely the AWS profile (this is defined when you configure your AWS CLI on your desktop) and the region where the cloud resource is to be created.
Finally, the third section defines the EC2 instance and specifies the AMI and the instance type required.
As shown in the example above, the Terraform declarative language is easy to understand and helps to keep all the cloud resources defined within the same project files. The full set of Terraform code to launch a working EC2 instance in the cloud can be found here:
https://github.com/enghee/terraform-aws-ec2-single-instance.git
Running the Terraform commands
The Terraform command line can be downloaded and installed from the following web page:
https://www.terraform.io/downloads
Once it is installed, go to the folder with the Terraform files (usually main.tf and other files with similar extension), and run the following commands:
To initialise Terraform and install the correct provider plugins:
terraform init
To build the cloud resources:
terraform plan
terraform apply
To destroy the cloud resources:
terraform destroy
As shown above, Terraform is not only easy to understand, it is also easy to run.
Conclusion
With Terraform, it is easy to start building your own cloud infrastructure. HashiCorp also provide tutorials to help you get started and reduce your learning curve. You can visit their tutorial at: https://learn.hashicorp.com/terraform.
Once you are familiar with the language, and ready to explore more of Terraform, go over to their Terraform Registry to see how others have written their infrastructure-as-code using Terraform and use those to create your own code to suit your needs!
References
[i] See reference: https://en.wikipedia.org/wiki/Terraform_(software)
