“It works on my machine” – evangelising continuous integration in HPC research

Research is an iterative process, and when writing code, we are often carving out a small area of functionality. Software version control is an essential part of this process, since we may want to try explore possible solutions, and revert back if the efforts did not bear fruit. It can also be key when we work collaboratively, as team members have a flexible, but standardised framework for combining work on the same system.

However, we have all encountered the old adage of “it works on my machine”, and to our chagrin nowhere else. Often that machine was our own 6 months ago. A part of the solution to this is containerisation, described in one of my earlier posts. Containerisation has its place for running full experiments, however for the development process of your systems, we need another piece of the puzzle, specifically related to our version control system.

Enter continuous integration

A mainstay of recommendations made by €150/hr devops consultants, continuous integration does have its place in successful projects, and is not daunting if you start small. It won’t solve all of your problems: write your code for you, help you achieve bigger lifts at the gym. What will it do?

In a continuous integration system (CI), there is a central server, which may be third party controlled, or in a small container on your existing infrastructure. This server has access to your version control repository, and will see new commits that have been pushed. You define a small script that will determine if your proposed version of the codebase is accepted. This script is checked into your repository too, so it can evolve over time as your needs change.

The script can be as simple as checking the code compiles properly, to more rigorous techniques, such as running your unit tests. The process of checking for new commits, and running the script is automated. The CI process is distinct doing these steps on your development machine because the CI starts with a clean slate with each job, avoiding pitfalls such as an inconsistent version-control state, magic dependencies, or other problems.

Extensions include setting up your CI to email the responsible team member for their build failing (because no one likes to nag), or running the CI script in different environments.

Adding a small file to your project that describes how to compile and test your code is enough.

Here is a an example. Assuming you are using git, with GitHub for your project, here we using the Travis CI system. In the root of your project directory, add a file called .travis.yml. Below, you can see the Travis script for a C++ project, which uses a Ubuntu environment, installs some packages, and then try and build the code:

language: cpp

env:
  global:
    # Ubuntu version
    - LINUX_DIST=trusty

matrix:
  include:
    - os: linux
      dist: trusty
      sudo: true
      compiler: gcc
      addons:
       apt:
         packages: &precise_latest_boost_packages
           - clang-7
           - clang++-7
         sources: &precise_latest_boost_sources
           - ubuntu-toolchain-r-test

script:
    ############################################################################
    # Build main and tests
    ############################################################################
     - export CC=clang-7
     - export CXX=clang++-7
     - git submodule update --init --recursive
     - mkdir -p _build
     - cd _build
     - cmake ..
     - make
     - ./run_tests

The CI system, in this case Travis, will have a web-dashboard, that will tell you the status of the CI jobs it is running. We can see the logs from each job to understand how our build failed, and perform a number of other useful tasks.

If you are using another version control hosting systems, then there are other options (e.g. GitLab CI, Jenkins, etc. Talk to you friendly local sysadmin). Annoyingly, the syntax can be different for each solution, however at a conceptual level things are the same.

This is a short post, trying to evangelise the use of continuous integration in research codebases. It’s a small amount of effort to get set-up, but can provide an immense amount of value, both during development, and later in a project’s life-cycle. If you’ve got a grad student, or recent new hire that looks too comfortable, get them on it!