The detour is the road

Okay, let’s put some context. I arrive in Nice and I have 2 months to develop and implement a few data post-processing routines in a plasma simulations code. The code currently stores files with data of thousands of particles that occupy an egg and the relevant information that can be extracted from there is not even a tenth of what it occupies, so my job is to perform the post-processing of that data within the code itself, so that its output is directly the information of interest. All good, all right.

Do you know how long it took me to start working on the post-processing routines? Almost a month! And you may ask: Julen, what have you been doing all that time? Easy. I’ve been understanding and fixing problems.

The first thing I had to do was to install the simulation code on the supercomputer where I would be working for the whole summer. Anyone who has ever done this process knows that this is usually a tortuous path full of incompatibilities and indecipherable error messages. But it was achieved.

Then we wanted to do some code scaling on this supercomputer. To put it quickly and badly: to see how long the code took to execute as we made it bigger and bigger (please, don’t kill me; if I’m not succinct, we’ll be here all day). You can’t imagine the headaches this brought, the behavior didn’t make sense! The code, with the same configuration, seemed to take random times to run.

How were the supercomputer nodes connected? What were their capabilities? What could be causing the erratic behavior of my code?

In the end we understood the behavior a little better: when I requested enough cores to have to make use of different nodes, the speed depended on the particular combination of nodes I requested to carry out my work. There seemed to be two branches: a fast one and a slow one, with consistent times in both cases. And why does this happen? Ah, who knows. It will be a mystery from now until the end of time.

By now it had been almost three weeks, between tests, research and, above all, many, many turns around the same thing. But it’s all right: let’s get to work.

The simulation code gives us the tensor of plasma pressures in different cells of the simulation box. We are interested in having the temperature tensor, and not in the code coordinates, but in local coordinates relative to the magnetic field direction.

Easy peasy. A little bit of formulation, a little bit of algebra, be careful with the units…

Everything seemed to work fine. What? Do you want me to put what I have calculated in a single file instead of having different files with the different components? Done.

Done? Difficulties come from where you least expect them. I had everything well calculated, but we wanted to save it all in a tensor form in a VTK file that we could open with paraview to visualize the data. This proved to be terribly problematic. How is the format correct? Why do I get everything right and the program doesn’t get the values right? Why is this all sooooo poorly documented? I just want to put in the numbers I have already calculated!

I have spent, I think, five times as much time trying to write things into files as I have calculating those things correctly.

I could go on telling you about my misadventures. I have implemented more routines and achieved interesting results. I will tell you about them in the last blog entry. But I didn’t come to talk about the results here. I came to talk about the process.

When you read about other people’s research and results, everything seems easy. We took this data, calculated that, and came up with these results. We wrote a program that did this and this, here it is… a lot of times you might wonder where all those months of work, all that time to do things that, a lot of times, don’t seem like such a big deal.

But that’s not the work. Research is hitting a wall over and over again, unable to find the door. Research is a labyrinth full of dead ends, where time and time again you have to retrace your steps. Even when you know exactly what you want to do, you can never imagine the problems that lie ahead until you actually do it. It’s spending hours and hours and hours and hours accomplishing nothing until you find the right way to do something and implement it in 10 minutes.

This may sound a bit disheartening, but there is a secret to this process. Learning. Every mistake, every unexpected problem, every little thing that you lose the whole day’s work to, is a learning experience. Do you know what I have learned about supercomputer architecture by scaling the code? What I have learned about data visualization and VTK format in my battle against writing data? Ooh, that’s what makes me now able to repeat what I’ve done all these weeks in a few days. Now I know more. But I know it because of all these little detours one encounters along the way. Those detours where we researchers spend almost all of our time.

So don’t get frustrated. Enjoy the journey. Even if you’re just going around most of the time. It may not feel like it, but you are moving forward. You are always moving forward.