Many researchers need to transfer large amount of data from one server to another server for backup/archival purpose or for processing in the target server.  Data transfer, or data move, speed will be important. The high-speed 100G network built between the National Supercomputing Centre Singapore’s (NSCC) and NUS aims to help researchers move huge dataset onto the NSCC’s ASPIRE1 system for analysing and processing. The 100G network is built with the 100G RoCE technology provided by Mellanox™.

Unfortunately, except for the native Remote Direct Memory Access (RDMA) tool that can drive and make use of the full bandwidth of 100G RoCE network by doing memory to memory copy. Currently, there are no other ready tools that have demonstrated such capability. In general, scp, rsync and sftp are the common tools that are used for the data transfer among research communities. As these tools are using the same network protocol for data transfer, hence scp is used to carry out the benchmark test.

Another unique tool “bbcp”, a free tool favoured by experienced users in performing large files transfer in multiple threads and hence improving the data transfer performance, is also adopted in benchmarking the data transfer performance over the high-speed network.

The setup of the test servers and the high-speed 100G network connection diagram are as below.

The servers H1 and H2 equipped with SATA hard disk are installed in NUS campus. The server H3 equipped with SSD hard disk (gives better disk IO performance) is installed in NSCC server room located in the Fusionopolis.

The network latency between NUS and NSCC is measured at 210 micro-seconds, mainly contributed by the 43KM fibre optical connection in between. The latency between H1 and H2 is only 0.6 micro-seconds, which can be ignored.

Test 1:  Data Transfer of One Large File Sizing from 10GB to 100GB

The first benchmark of data transfer is to test the data transfer speed by moving one large file from server H1 and H2 in NUS to H3 in NSCC. The file size is 10GB, 40GB and 100GB, closing to the usage disk space available on the servers. The data transfer outcomes are tabulated below. The data transfer speed is measured by the ratio of file size against the time taking to complete the transfer.

• For data transfer using scp: the speed is about 240+ MB/sec using single scp session.

• For data transfer using scp: the speed can reach about 380 ~ 450 MB/sec when running two scp concurrently.

• For data transfer using bbcp: the speed is about 1.1 ~ 1.2 GB/sec for file size of 10GB and 40GB when using two or more threads.

• For data transfer using bbcp: the speed is only about 330 ~ 565 MB/sec for file size of 100GB. There is an obvious drop of speed is observed during the data transfer course.

Test 2: Data Transfer of Many Files with Total Size of 100GB

The second benchmark test is to emulate a realistic scenario to measure the data transfer speed by moving many files from one server to another using tools scp and bbcp. With the available 100GB usable disk space, the test was performed at two different datasets:  200 pieces of 500MB files and 1000 pieces of 100MB files.

The outcomes of the data transfer speed are summarized in the below two tables.

• For data transfer using scp: the speed is consistent at 190+ MB/sec for data transfer from H2 to H1 and 240+ MB/sec for data transfer from H2 to H3.  This implies that the data transfer speed for scp tool is majorly determined by the distance or the network latency between the source and target servers, as the shorter distance or smaller latency gives higher speed.

• For data transfer using bbcp: the speed is about 320~350+ MB/sec for data transfer from H2 to H1 and 640~920+ MB/sec for data transfer from H2 to H3.  This implies that the data transfer speed for bbcp tool is majorly determined by the target server disk IO performance, as the better disk IO performance on the target server gives higher speed.

The data transfer performance was benchmarked on the recently established high-speed 100G network by measuring the data transfer speed using the available tools scp and bbcp for both single large files and also many files. The benchmark results demonstrate that scp tool in general gives a consistent speed for the range of 190 to 240 MB/sec; while bbcp tool gives a higher speed ranging from 350+ MB/sec to 1.2 GB/sec. The actual data transfer speed could be affected by many factors, including but not limited to, file transfer tools, size of file(s) and number of files to move, latency between the source and target servers, and target server disk IO performance, etc.

The team is open to explore, test and implement other potential data transfer tools that can demonstrate better performance in moving data through the high-speed 100G network.