MY JOURNEY IN THE JOB REGRESSIONS

Hello everyone, I hope you are all well and with your loved ones during the pandemic period. After SoHPC’s rapid introduction to my life, I set out to advance the project. There were concepts I know a little, but I was motivated to deal with artificial intelligence.

The Beginning of My Journey

As I mentioned in my previous post, me and my teammates Francesca were developing an algorithm to determine the time limits of the jobs people submitted to supercomputers. In this way, jobs won’t be thrown to the end of the queue due to back-filling optimization. If I briefly explain the back-filling optimization:

When the workload manager (SLURM) receives long-term jobs, it controls the nodes to execute them. SLURM manages time usage efficiently by sending short-term jobs to idle nodes. A simple visualization of the back-filling optimizations is made in Figure 1. However, because users enter the time limit much more than the elapsed time, SLURM considers these jobs as long term jobs and assigns the job to the end of the queue.

My Travel Pack

We have provided a database of the job batches of the supercomputer of Hartree center to develop the regression algorithm. The data provided are nearly 2-year job distribution data of 5 users. As seen in Figure 2, some of these users are people who determine the time limits of their jobs reasonable, and some of them give their limit times higher than the elapsed time. With this feature of data, we had to make an appropriate prediction for both types of users.

 Where am I on The Road?

When the project started, our motivation was to complete the project with the least extra resources. In this way, users will be able to use the project without using much extra resources. For this reason, I prepared two regression model from scratch and a regression model created with sklearn.

What Awaits Me on the Continuation of Journal?

We have achieved considerable efficiency in time limits with our predictions, but our current goal is absolutely no underestimation in the time limit predictions.

Apart from the paperwork, we can learn to apply a regression similar to our short-term jobs in our own daily life. In this way, we can create extra time for our hobbies. For example, I like to play my guitar when I wait for things to happen on the computer. So, what are the back-filling optimizations you do in your daily life?