Enabling lattice QCD simulations on GPUs

Project reference: 1805

Lattice Quantum Chromodynamics (lattice QCD) can solve and predict physics in the low energy regime of particle physics where analytical methods such as perturbation theory are not applicable. Realistic simulations of QCD are carried out using Markov Chain Monte Carlo for producing configurations with lattices of billions of degrees of freedom. High Performance Computing is therefore crucial for calculating observables within lattice QCD that can be compared to experiment and/or provide insights to searches for physics beyond the Standard Model.

The project will target optimization of a lattice QCD simulation code to use GPUs. Namely, the tmLQCD package will be developed to call the appropriate functions of the GPU library QUDA. The PizDaint supercomputer at CSCS in Switzerland will be used to develop, benchmark and scale the implementation.

Analysis codes using QUDA have already been developed and are running on PizDaint via projects secured by the local group. Enabling simulation codes via this project will mean that both major components of any lattice QCD calculation will have been enabled on GPUs, thus contributing major efforts in future-proofing these community codes for Exascale.

The summer student will obtain access to this machine in order to carry out the project, which includes optimization for lattice QCD simulations with Nf=2 twisted mass fermions. Here the challenge is to optimize the Hybrid Monte Carlo (HMC) setup such that the precision is high enough to yield a high acceptance rate while keeping the computational costs moderate. For the HMC, a nested fourth order integrator, in combination with Hasenbusch mass preconditioning will be used. The challenge is to identify an optimal set of solvers for each of the multiple Hasenbusch mass terms. This involves scanning the parameter space of Hasenbusch masses while measuring the time to solution of the solvers to obtain the combination that yields the smallest time-to-solution.

Analyses on PizDaint have enabled the calculation of the so-called σ-terms, observables that probe interactions of dark matter with nucleons and can therefore guide dark matter searches in experiments. The project will enable the next level of such measurements by porting the simulation component to the PizDaint architecture. Source: https://www.cscs.ch/science/physics/supercomputers-on-the-trail-of-dark-matter/

Project Mentor: Giannis Koutsou

Project Co-mentor: Constantia Alexandrou

Site Co-ordinator: Stelios Erotokritou

Learning Outcomes:
The summer student will perform runs on PizDaint, a PRACE Tier 0 system, and analyse solver performance on GPUs. The summer student will be introduced to basic concepts of Monte Carlo algorithms in lattice QCD.

Student Prerequisites (compulsory):
Undergraduate degree in Physics with grade above average and good programming experience.

Student Prerequisites (desirable):
Knowledge of Theoretical High Energy Physics; Experience with parallel programming, Experience with GPU programming

Training Materials:

• Lattice Gauge Theories – An Introduction, Heinz J Rothe,
• World Scientific Quantum Chromodynamics on the Lattice, Gattringer, Lang, doi 10.1007/978-3-642-01850-3, Springer-Verlag Berlin Heidelberg
• Lattice Quantum Chromodynamics, Knechtli, Günther, Michael and Peardon, DOI: 10.1007/978-94-024-0999-4, Springer Netherlands

Workplan:

• Week 1
  During the first week, the summer student will attend the PRACE SoHPC training week.
• Week 2-4
  From the second to the fourth weeks, the summer student will perform some first runs on PizDaint for familiarizing with the code and the supercomputer environment. This includes, compiling of the software, running the different solvers, and carrying out an introductory tutorial to multigrid solvers and HMC simulations in lattice QCD by experts from the local group.
• Week 4-7
  In the fourth to seventh week, the summer student will evaluate different configurations of a nested fourth order integrator by measuring the energy violation during the molecular dynamics update. By varying the step-size and the Hasenbusch masses, an optimal cost for a given energy violation can be obtained. Moreover, possible parameter corrections can be implemented in the calls to the solvers.
• Week 8:
  Refine results and prepare final presentation, hand-over to local researchers.

Final Product Description:
The final results of the project will be an optimized Nf=2 twisted mass clover HMC simulation code that can be deployed on PizDaint. A report will include graphics to visualize the performance improvement obtained by the use of GPUs. Diagrams will also be used to visualize the parameter space search for obtaining the optimal set of HMC parameters.

Adapting the Project: Increasing the Difficulty
The intensity/difficulty of the project can be increased by comparing different integrators, such as between second and fourth order integrators. Moreover, the optimization can be extended to a GPU machine with different communication architecture for comparison.

Adapting the Project: Decreasing the Difficulty
The intensity/difficulty of the project can be decreased by reducing the QUDA functions called to the bare minimum (i.e. only the solver).

Resources:
The student will obtain access to PizDaint where the local group currently carries out analysis of lattice QCD configurations.

Organisation:
The Cyprus Institute – Computation-based Science and Technology Research Center