Building a supercomputer in a cloud

With my second week ending here at SURFsara, I’ve learned quite a bit about the background of my project and those insights have been very interesting, so I thought I’d share them with you!

On a usual supercomputer, networks and file systems are shared. Users can make connections from inside the network to the outside, and file privacy is handled with the POSIX file system permissions, determining which users can read, write or execute a file. However, if the permissions to a file are set wrong, unauthorized users can access the respective data.

If your files just contain cryptic radar measurements, you probably don’t care too much who gets their hands on your data. But if you are researching genetic makeup on Dutch government-owned confidential data, that’s different. This is why SURFsara is looking to build a very secure HPC platform for linking, analyzing and processing sensitive data. They use PCOCC (Private Cloud on a Compute Cluster) to create virtual private clusters on their supercomputer Cartesius. To create these virtual private clusters, PCOCC makes use of KVM to create virtual machines and ties them together to a cluster using Open vSwitch for network visualization and SLURM for job scheduling (you don’t have to click on all the links in order to understand this post, don’t worry). The clusters are highly customisable, so for example, if you want to prevent users from taking the data they’re working with outside, you ensure that they can only access the data through a VPN which does not allow any access to the outside world. After the research is finished, the virtual cluster is destroyed and all data is cleared. This prevents intentional, but also unintentional data leaks.

My project for this summer is about eliminating another (minor) security risk for the platform: the shared disk space. If the disks used by the virtual private cloud were encrypted, data can’t be leaked when someone unauthorized gains access to the files. So I’ll be working to add support for encrypted volumes to PCOCC (it’s open-source!).

These past few days, I’ve been making steps toward building my test environment. I can’t really just play around with encryption on Cartesius, so I’m building myself a mini-supercomputer on which I will then run PCOCC. But unlike Caelen’s Wee-Archlet, my setup is a bit… cloudier. SURFsara has an HPC-Cloud which runs on physical hardware located somewhere in their datacentre (next to Cartesius). In this cloud, you can create virtual machines. These virtual machines can have more or less whatever amount of CPUs and storage that you like and run your operating system of preference. The amount of CPUs and storage are, however, somewhat limited by the underlying physical infrastructure. Letting users work in virtual machines instead of, for example, on Cartesius, means that users have more freedom in what they want to do and need less technical expertise, too. If you’re working on Cartesius, you open a terminal, log in to the supercomputer and execute some batch scripts. In a virtual machine, you get a graphical interface with an operating system you feel comfortable with, you can install software, open tools, work with what you’re used to and you can have root rights. So in this HPC-Cloud, I set up two virtual machines running CentOS (I chose randomly) and those two are my new mini-supercomputer. More or less. These two virtual machines act as two nodes of my supercomputer. By the way, Cartesius has about 2000 nodes. To make the virtual machines act as a supercomputer, I had to configure the communication between them and set up SLURM, the job scheduler. For storage, I configured a third virtual machine as an NFS-server which you can think of as a disks you can access only through your local network.

The next step will be installing PCOCC on top of this “supercomputer” made out of virtual machines running in a cloud, so that PCOCC can create its own private cloud with virtual machines running inside it. If/when that works (fingers crossed!), I will start playing around with encryption. See you then!