Posted on by Thizirie Posted in Blogs 2019, PRACE Summer of HPC 2019 — No Comments ↓

Ahoj! [pronounced: Ahoy]

Here is the essence of HPC: if someone tells you “congratulations you solved the problem”, that person is not HPC. All solutions should be met with “Congrats, now how do you solve a problem twice as big, or use a 2x computer to solve in half the time?”

Someone told me this in my last post’s comments, and it is definitely something I believe in. Thank you Rich for the advice.

Now, welcome to this second part of the HPC journey!

If you read my first article you already know how amazing my first week was in Bologna, Italy. (If you haven’t read it yet go check it out just here so you can know more about me and how this summer school started.)

Now let’s move to the core of this post!

Two weeks ago, I flew to Vienna then took the bus till Bratislava where I started my very exciting project which is about… High Performance Machine Learning! So I will be working every day here in the Computing Center of the Slovak Academy of Science on it!

  • Ciao Bologna!
  • Ahoj Bratislava!

Machine Learning

Let’s start briefly with Machine Learning; one of those abstract fields we hear about, but do not really know what it is. Some people can take a quick look into the subject and have an insight about how cool it is, others chose to dive into this thrilling field in order to figure out how it works and learn to use it efficiently. The third kind of person includes the ones who let down their researches about ML because of maths, the huge amount of information. But let me tell you something; these people have more in common than what you think! First, they (we) all still have a lot of things to learn (there is no edge to improvements and knowledge) but also… they are all looking for efficiency.

So what is Machine Learning? (a brief introduction)

  • A subset of artificial intelligence.
  • It is about studying algorithms Based on statistics;
  • Needs lots of data!
  • Its goal is to derive meaning from the data you provide, but without giving explicit instructions (about correlations between some variables, for example)

– Watch this video for more.

And here is the entrance of the very large SAS site.

Now, this is where magic happens! And by magic, I mean HPC… 

Requiring huge amounts of data means that Machine Learning needs lots of resources for the processing and analysis of these inputs, and you may already be able to see what I am trying to point out… How critical time is for these problems of course!

The more data you have (to train your ML algorithm), the better it may perform (although, it depends on other parameters… We’ll see that later on other posts!) but this means that you need more time, more energy and more resources (computers or CPUs, cores, GPUs) to do everything.

So, with HPC we can reduce this runtime and make sure that our algorithms are able to run and process everything within an acceptable time. But you may ask me, how do we do that?

The answer is simple… Nah, just kidding. Otherwise there wouldn’t be a whole summer school dedicated to this matter and so many people working and trying to gain more in time/energy every day (by reducing the runtime, etc…).

The most useful things I used on my first week here in Bratislava! No programming till I understood the nuts and bolts of my project!

At this point you know what I will be dealing with this summer. There are only few points missing like the ML algorithm I am talking about. There are actually many of them. Nevertheless, I will only talk about Gradient boosting trees since I’ll work on this one.
In the future blog posts, we will delve into the aforementioned algorithm and how we are going to optimize it with MPI and GASPI (the HPC tools we were talking about). But don’t worry, we will talk about HPC before that and I will try to explain the concepts!

Now, I will leave you with my work plan and I look forward to hearing your feedbacks. See you next week for a less technical post where we will talk about trips, museums and food!

Weekly schedule of the summer. Very excited about every step! Created using https://www.visme.co/

Posted on by Rebecca Lait Posted in Blogs 2019 — 14 Comments ↓

As a vegan, I am undoubtedly familiar with protein and how important it is to our survival – even more so now that I have been studying it for my PRACE Summer of HPC Internship. More specifically, I have been researching KRAS, a protein that is an oncogenic mutant – with an oncogene being something with the potential to cause cancer and a mutant being an organism resulting from an instance of mutation.

Since my introductory blog post last week, I have learnt so much! Our training week at CINECA consisted of learning an array of useful skills that were all new to me – but which I hope become much more familiar as I progress through this internship. The week started with an introduction to Linux and progressed to parallel programming with MPI and also to OpenMP later on. The training period in Bolgna then finished with an introduction to GPU‘s and to some visualisation programs called Paraview and Blender. I very much enjoyed this visualisation aspect after having concentrated on pure coding earlier on in the week- especially because I enjoy making and looking at scientific simulations. During our visit to CINECA, we were also able to visit the supercomputer there.

This is a photo of myself and the other 2019 PRACE Summer of HPC interns and coordinators. This was on our first day at CINECA, Italy. Source: CINECA.
  • This photo shows a visualisation program we used during our time at CINECA.
  • This photo of me was taken during the tour of the supercomputer at CINECA, Italy.
  • This photo shows a second visualisation program that we used in Italy.
This photo is from the supercomputer at CINECA, Italy. I was fascinated by the sheer number of cables required for each of the machine units!

The first institution I was introduced to after arriving in Greece was the Biomedical Research Foundation, BRFAA, where the majority of my time will be spent. My first task was to write a report on the various inter- and intramolecular forces found in proteins and in protein-ligand complexes. Proteins consist of multiple amino acids that are all held together via peptide bonds. These proteins can fold and hence form an enormous range of differnet structures. It is these configurations of the chains of amino acids that helps to determine the protein type.

After finishing this report, the second assigned task was to complete an online tutorial in order to learn about NAMD, a parallel molecular dynamics code, and molecular dynamics simulations. This tutorial gave such useful insight into the world of simulations and introduced me to the range of possibilities available with this program, and others like it.

This is a video I made on my first day before I arrived at the BRFAA in Athens.

My third week of the Summer of HPC internship was located at GRNET, where the supercomputer, ARIS, is managed. The week consisted of learning a multitude of new skills and the continued support and guidance increased my confidence when using new software. We completed tutorials in Gromacs and were taught about important parameters and initialisation steps when creating and studying proteins. In addition, we were also lucky enough to be able to visit ARIS, the supercomputer, itself. Thanks to PRACE Summer of HPC, this visit made it the second supercomputer I had visited in just three weeks! A photo of me visiting ARIS is the featured photo for this blog!

Next week, from Monday 22nd July, I will return to work at the Biomedical Research Foundation. The week will consist of learning more about my final project and becoming more familiar with the molecular dynamics simulations that I will be working with. This will be the final training week, before officially starting work on my final project and I am very excited to continue learning more about KRAS, the protein I will be studying.

  • This photo shows the Biomedical Research Foundation building
  • This photo shows the impressive structure of the inside of BRFAA
  • The many floors of the BRFAA house multitudes of interesting research

I am very much looking forward to the upcoming weeks and would love to share my journey with you. You can stay updated with my blog posts throughout the summer and I would love to hear what you are doing this summer as well! Is it related to supercomputers too or maybe another area of study? You can leave comments below as I would love to read about all of your plans and projects!

Posted on by Antonio Miranda Posted in Blogs 2019 — No Comments ↓

1.3 billion years ago. Two blackholes which had being dancing, spiraling inexorably, for millions of years, finally merged in a distant region of the universe, rippling space-time .

December 10th, 1909. Around 1000 guests are reunited at the Stockholm City Hall including a 35 years old italian and the King of Sweden. A couple of hours earlier, this Italian scientist, Guglielmo Marconi, had stepped forward to receive the Nobel Prize of Physics. He and many other scientists had been working on an invention that changed our world: radio. As many other scientific breakthroughs, it proved its value during World War I. Armies did not need wagons pulled by horses to carry cables along enormous distances to communicate any more: we had entered the wireless era.

Guglielmo Marconi. Source: Library of Congress, US. Deposited in the US Copyright Office before 1924.

September 14th, 2015. Echoes from the merging of that two old black holes were detected at LIGO Observatory. It was the first time that we, humans, came to detect gravitational waves -disturbances in space-time curvature. This discovery not only confirmed one of the most fascinating predictions of Einstein’s gravitational theory but also opened a completely new window to the study of the universe. Many have said that, until yesterday, we were just seeing the universe while today we are also capable of hearing it.

Today, Italian scientists are studying gravitational waves and MARCONI is involved this research. How so? We have realized in the recent years that, to perform this and many other experiments -such as the Higgs Boson discovery-, huge computational resources are needed. US and China are the leading the race of supercomputers but several European countries have partnered to work together on building competitive supercomputers and created the PRACE partnership. One of these countries, Italy, is naming his supercomputers after eminent scientists and the biggest one of them, #21 in the world, is called MARCONI.

Two technicians working at MARCONI. Source: CINECA Flickr account. Permission granted by author.

Under the umbrella of such an amazing institution, 25 European late degree and master’s students are developing 25 different projects in 12 European supercomputers. First stop, Bologna -home of MARCONI supercomputer-. For one week, we learnt how to use key technologies to work with supercomputers such as OpenMP, MPI or CUDA. All of this in situ, testing our codes at MARCONI’s younger brother, GALILEO. After this training, we departed to our destination cities. In my case, Athens.

Group picture of Summer of HPC participants and coordinators. Source: CINECA. Permission granted by author.

And that is how, on July 6th, I arrived in Greece: more than 2000 islands, excellent food, astonishing history and one great supercomputing institution: GRNET. There and for the next 8 weeks I will be working together with a team from BRFAA on finding more effective drugs. For a drug to be active, its molecules must bind tightly and selectively to a target molecule in our bodies. Once they are bound, they generate the desired effect in our health. Consequently, to find effective drugs, we evaluate how strongly a drug molecule binds to its therapeutic target.

In fact, we are evaluating the binding strength of many drugs and select the one that binds more strongly. This one has a stronger effect on our health and, consequently, taking a fewer amount of it the same therapeutic results are obtained.

To find out the binding strength, we use computer simulations. These simulations take the structure of the drug (how the atoms are located) and its target. Then, using fundamental physics laws such as Newton’s equations of motion, they calculate how good a linkage is. Since thousands of these simulations are needed, supercomputers are needed (nop, sorry, your Acer laptop is too weak for this).

The most accurate technique to simulate and compare several bindings is called Free Energy Perturbations. In FEP, the key magnitude is the amount of energy that can be extracted from the system (without irreversibly changing it). This magnitude is called Gibbs Free Energy. And what we call “a system” is the combination of a drug, a protein and water around them. Thus, we compare how this energy changes when we modify the drug of the system. The bigger this change, the better the new drug is. Hence, my project compares many modifications of a drug used in cancer metastasis treatments and finds the one which binds better to the target. That one could be more effective, reducing the dose needed. The final goal is to compare simulation to experimental results to prove FEP is a valid technique to create better drugs.

Hemoglobin structure. Source: own.

Keep an eye on my posts (tag “Antonio Miranda”) to share this journey with me!

Posted on by Allison Walker Posted in Blogs 2019 — 4 Comments ↓

It’s the end of my second week at SURFsara, and I’m off to a flying start. As much as I would love to get into the nitty gritty of my project and all of its technical components, I will save that for a later post. Instead, I want to give my dear audience a quick update on the context of my project: Birds. 

The key clients for this project are ornithologists from the e-Ecology department at the University of Amsterdam. This department is keenly focused on accurately predicting migration patterns for different bird species, and across different migratory seasons. These ornithologists use two data sources when conducting their research on these patterns: 

  1. GPS trackers (little bird backpacks), and 
  2. Meteorological radar data (typically used for weather forecasts, but also capable of identifying objects).
Falcon with GPS tracker

The focus of my project is on improving the way in which these scientists are able to load and analyse the latter. As I was briefed on this project, the question I kept coming back to was: why? Why is it so important to analyse and predict bird migration and the influence of weather on these patterns? Well, it turns out there is a surprisingly varied number of reasons. 

Fact sheet:

  • Bird strikes are the second leading cause of aviation-related fatalities.
  • More than 400 people have died in the last 30 years as a result of bird collisions with aircraft.
  • Airports all around the world hire ‘bird controllers’ to prevent bird strikes.
  • About 200,000 birds are killed every year in collisions with windmills.
Wind farms can be very dangerous for migrating flocks

The GPS data collected from birds shows that these animals are not consistent in the timing or route chosen for migration. It is clear that the weather has a direct impact on how these migratory patterns change, but the nature of this relationship requires further analysis. Some of the real-world implications of accurately predicting bird migration are:

  • The aviation industry as well as Air Forces across the globe adjusting the location or timing of flights.
  • Wind farms stopping or slowing their wind turbines when a large number of birds are flying through.
  • Switching off lights in tall buildings and communication towers, which have been known to confuse birds flying at night.

Ok, that’s all for now folks! Hope you enjoyed the sneak beak (sorry, not sorry) into my summer project 🙂

Posted on by Caelen Feller Posted in Blogs 2019 — No Comments ↓
MPI Ring Demonstration on Wee Archlet

So, I’m sure you’re wondering what the above video shows. It’s the product of my first week of work here in Ediburgh, and is called a “Wee Archlet”. It is called this because it is a cluster of Raspberry Pis, much like Wee Archie, but smaller. Can they actually talk? Yes, but only to each other, and only if they use a system called MPI (Message Passing Interface). The arrows shown tell us when a Pi is sending or receiving a message from another Pi. Here, they do so in a “ring”, which I’ll explain below. I’ll also explain how I built this, how the animations are shown, why it’s useful, and share some nice photos of Edinburgh!

Building a “Wee Archlet”

A Wee Archlet – Several Pi Joined Together

The first task I was given when I arrived in EPCC was to build an Archlet, so I understood exactly how Wee Archie was put together. Setting up a Raspberry Pi is much like setting up any Linux computer – you can do it graphically, or over the network – I won’t go into that. The interesting parts of the configuration are how they share information, and how they show it.

Setting up MPI on a Raspberry Pi is fairly easy, and allows machines to start talking as soon as you have them on one network, and they all know where to find each other. You can see below the small “Ethernet switch” I use to connect them all to a local network. An interesting aspect of the setup that is very convenient is a shared drive. Raspberry Pis all have their own SD cards to store files on but it’s possible with Raspian (the operating system they run) to share a drive over the network. This means they can all have access to the same storage space as fast as the network allows, and means you can easily share data, programs, and output between Pis.

EPCC have a very good guide on how to build your own, if you’re curious about more details!

  • A Single Raspberry Pi
  • Router Setup

The Animation Server

Running MPI code on the Archlet is as easy as running it on any other cluster once it’s set up, but showing how the nodes communicate is not a typical thing to want to do. My LED panels are Adafruit LED backpacks, and while you can communicate with them in C, it’s far more pleasant to do so via Python. Adafruit provide a library handling the hardware end of things, such as bus communication, which you can find here. Then I can just talk to the Python script from C, making it as simple as calling a slightly modified MPI function!

Wee Archlet Idle Animation

With this Python library, I can do simple things, such as this idle animation, by saving the animation as a sequence of 8×8 images of only ‘0’s and ‘1’s, then loading them each in turn and turning on the LEDs corresponding to ‘1’s and off the ones corresponding to ‘0’s. But handling the complicated types of animations necessary to show even a simple send and receive required a more complicated solution – my animation server.

On each Pi, I run a python script which starts a web server, built using Flask, a lightweight server framework for Python. Whenever this web server gets a connection with the necessary data, it will tell a seperate script (the display server) to play the correct animations. The web server will also take care of talking to the servers on the other Pi’s if it is necessary. For example, in the send and receive animations, each send has to wait for a matching receive to start and vice versa. If this isn’t clear, I’ll describe it more below.

The display server runs on a seperate process to the web server, and it will queue all the requested animations and processes up in memory, and work through their playback by converting them into single images, loading them onto the LED, and moving onto the next one.

Once it’s finalised, I’ll revisit the animation server in a later post, and go through the more complex, smaller details. For example, the display server need to be able to always receive more animations from the web server, so you have to use one of Python’s many concurrency solutions. I’m currently using processes, but that may change as need demands. I also might change the queue based display server to one that actually reads the animation from a static file on the SD card, allowing for more complicated sequences of animations, replays, etc.

Making a Ring

MPI Ring Demonstration on Wee Archlet

Just to be sure it’s clear, I’ll run through what’s happening in that first video. I’ve embedded it again here, for reference.

This is a visualisation of a simple C program using MPI, using the animation server discussed above. In the program, the bottom Pi will send a message, just the number “1”, to the one above it. Then that Pi will pass it on to the one above it, and so on. When you reach the final Pi, it will pass the message back to the first one, completing the “ring”.

This may seem useless, but imagine you need to add together numbers stored on each Pi, the results of some computation run in parallel. A simple way of doing this is this ring – add your own number, pass on the sum. When it gets back to the start, you’re done! There are more sophisticated ways of doing this, but I’ll get to them in a later post.

The C program is fairly standard among MPI demonstrations, but instead of just sending or receiving a message, it instead calls my special wee_MPI_Sendor wee_MPI_Recv methods, which first will send an HTTP request to my animation server. I do this using libcurl, a standard HTTP client library in C. Then the server processes the incoming animations as discussed above.

Goals

While this is a wonderful start, there’s still lots to do! My next goal is to make my animation server support animations for every MPI operation I think will be useful, moving beyond simple blocking sends and receives (ones which wait for a response).

I’ll flesh out the C library which communicates with the server, and then all that will be necessary to visualise the communication an existing simulation will be to use the correct functions. I hope to have a full family of simple demonstrations like the ring above that help build the animation “vocabulary” and explain basic parallel computing concepts.

The first “proper” simulation I’d like to visualise is a wave/coastline defence simulation as I mentioned in my previous post. The main issue is choosing what to visualise, as if I just showed every communication I did, as I have here, it would take hours to finish playing! I’ll break down the concepts involved and tasks necessary in a post when I get working on it. Hopefully I’ll have time to try visualising other simulations too!

Personal Update

Edinburgh is beautiful (see pictures below). Over the last two weekends, I’ve made a start on exploring the city with the other SoHPC students. Here are just a few of the locations seen, but I’ve also spent a lot of time in city’s various museums. While I can’t post photos, I would really reccommend visiting the National Museum of Scotland if you ever get the chance, it’s a wonderful collection of history, art and science. We’ve also played a great deal of the board game “Sushi Go: Party”, and visited a board game café!

  • Botanical Gardens
  • Train Station
  • Arthur’s Seat

Also, I’ve been playing a lot of Table Tennis with Benjamin Huth in my accomodation’s rec-room, and while I’m still far outstripped, I’m hopeful I’ll improve enough by the end of the summer to hold my own! More on this as it develops…

Posted on by Benjamin Huth Posted in Blogs 2019 — No Comments ↓

Welcome to my first blogpost about my project in Edinburgh! I have to admit, the heading is quite dramatic, and I’m not sure if I can live up to your expectations while talking about some obscure computing topic, for which the Wikipedia article mostly consist of a big warning box about content issues. But I’ll try my best!

Maybe I should first describe my project in a few sentences: EPCC (the super-computing institute in Edinburgh) have created a library for writing parallel code. Its idea is that the programmer does not care about the actual transfer of data between the processes, they just define tasks which are automatically executed if certain conditions are fulfilled. My task in this remaining six weeks is to replace the messaging-backend of the library from MPI to an API called GASNet.

GASNet and active messaging

GASNet (Global Adress Space Networking) and MPI (Message Passing Interface) are both standardized APIs for communication between processes. But that is about everything they have in common. One finds out the first big difference very quickly: For GASNet you can’t find help on the internet. No code snippets in stackoverflow, no threads in obscure IT-forums. Apparently, nobody uses this API or at least nobody has questions about it. This was surprising to me, since I had many questions and it has a very different approach than MPI.

Picture of the "thief-analogy" for active messages.
A typical day for a process in a supercomputing environment. All used images are license free content.

This approach is called “active messaging” (AM). To illustrate that, imagine your supercomputer is a small town. You are the local process, and in your house you have your local memory in a chest. With MPI, another friendly process asks for some data, and you call a postman, who delivers your parcel to the right adress. In contrast to that, the active message is like a thief. He goes right into your local memory, picks up what he needs, and disappears. You can only detect him by checking, if he has modified your data, and you also cannot stop him from doing this. He is truly a master thief! Maybe he also leaves things at your chest, but that’s the point where the picture becomes slightly unrealistic.

Of course, if you write your code, there are no active messages who steal your data. I just want to point out that there is no kind of recieving function call on the target process like with MPI. The message is called active, because it carries its own executable function with it, not just passive data.

Scheme which describes request- and reply during an active message.
Scheme of AM usage. After the message arrived, the program flow of process 1 is interrupted, and the request-function is executed. The user can decide if he wants to send data. Sometimes the reply is optional.

Why should one do such things? You can understand it better, if you take a look at the hardware-level and how communication is implemented there, as it’s in nearly the same way! The process sends its message through the wires, and then in the target process there is a hardware-interrupt, so the execution of the normal program stops, and commands necessary for receiving the message are executed. Then the execution of the main program continues. Our hope is if you emulate this also on software-level, you can get better performance. But more about that in later posts…

Some thoughts about Edinburgh

Now, after you survived the dry part about technical stuff, I want to finish the post with some impressions of Edinburgh, a quite interesting city (but which city is not interesting?). But not everything is fast and simple there. For example, some streets change their name every 200m, if you waited for the green light on each crossroad, you would need twice the time Google maps suggests. In particular, Google maps isn’t very helpful, especially if you’re searching for concurrency-exchange offices (which seems to be a very volatile business here).

Roof-top bar in Edinburgh with view to the castle.
Conclusion of a working day in a roof-top bar below the castle of Edinburgh.

But apart from that, I really like it! All people we met, especially at EPCC, where very nice and friendly. In about three weeks “the Fringe” (a huge art festival) begins, so the whole city is preparing for it, with stages assembled in every corner. My site-coordinator told us, that the population of Edinburgh will increase from about 400,000 to 1.5 Million people during this time. I’m sure August will be a totally different experience!

Posted on by Arsenios Chatzigeorgiou Posted in Blogs 2019, PRACE Summer of HPC 2019 — No Comments ↓

The first week of July was the training week, that took place in CINECA, a research center in Bologna, Italy. During that time, I got the opportunity to:

  • eat a lot of pasta and pizza
  • drink beer in a very cheap bar next to our hotel
  • learn about and meet most of the High Performance Computers (HPC) of CINECA
  • learn about and meet most of the High Performance Persons (HPP) of SoHPC
The HPC MARCONI cluster next to the HPPs SoHPC cluster.

A (relatively) bold statement

During the training sessions in Bologna, we learned that there are two popular ways to get the most out of an HPC cluster. Here, I will try to prove that the same two ways apply to an HPP cluster.

Definitions

In order to support such claim, I have to define some things.

An HPC cluster is an aggregation of computer components, which when run in parallel and interconnected, may have multiplied performance than the stand alone components.

Arsenios Chatzigeorgiou after the training week, i.e. week#1.

Interconnected HPC is called cluster and the things that interconnect are called nodes. Each node is roughly a single computer unit just like the one you probably read this post (unless you have printed it on paper, which I would discourage you from doing it again for environmental reasons).

An HPP cluster is a group of people that have a shared target and are willing to work together in order to achieve this goal.

Arsenios Chatzigeorgiou after the thinking-of-the-training-week week, i.e. week#2.

Interconnected HPP is called group and the things that interconnects are called participants of this group. Each participant is a human organism just like you (unless you are some kind of replicant which is fine, but then the HPP would be HPC again).

Methods

The two popular ways to run an HPC is message passing which is implemented with the MPI standard (Message Passing Interface), and shared memory which is implemented with the OpenMP API (Open Multi-Parallel).

MPI and OpenMP in HPCs

Using MPI, each node implements different tasks independently (heterogeneous system) and the nodes communicate via triggering signals. The communication is achieved with the messages. It is a flexible and expressive method.

In OpenMP when a complex task arrives, a fork path might emerge. The task can be divided in many similar subtasks that each node will be assigned to (homogeneous system). The communication is achieved with the shared memory among the nodes. It is easier to program and debug.


The two different ways to run an HPC.
Thanks to Nicola Spallanzani for allowing me to use his images.

MPI and OpenMP in HPPs

Some groups might perform better when each participant of the group is having independent tasks (heterogeneous) and is isolated in its own place. In the procedure however, it is crucial to have an efficient message passing system to support the communication between participants.
Such method demands persons who can take initiatives and be able to adopt in dynamically changing tasks.
This is MPI in HPPs.

Other groups, perform better with a leader that organizes all the participants. A difficult task, will be allocated into many participants who will perform similar job (homogeneous). What each will do will be directed by the central participant who organizes them.
Such method is easier to manage for an organizational person. However, flexibility is limited and many participants are demanded for better performance.
This is OpenMP in HPPs.

Denouement

Each SoHPC participant is going to work on a group of HPC, on a specific project in any of the aforementioned methods. In my case, in LECAD lab in University of Ljubljana, this first week we have started using the MPI method for visualizing a gyrokinetic simulation, both for the HPC application and for the HPP group organization.

So, each of the SoHPC participants are going to operate using their own method, but we all going to have the shared memories we gained in these 5 days in Bologna.

This year’s SoHPC group in front of Fontana di Nettuno in Bologna’s main square.

I hope I helped you learn something today. In the next post I am going to describe what is a gyrokinetic simulation, what I am trying to do here in LECAD and how I plan to do it (hopefully I will have figure it out) .

Disclaimer: the “High Performance Person” joke was made initially by Matteo Stringher and Antonio Mirada.

Posted on by Khyati Sethia Posted in Blogs 2019, PRACE Summer of HPC 2019 — No Comments ↓

Pozdravljeni iz Slovenije!

Today I want to share my experience with you about the training week of SoHPC in CINECA (Supercomputing Centre for Scientific Research in Bologna, Italy). The week started with meeting new people from different countries. We attended the training sessions from morning 9:00 AM to 6:00 PM. Though we were somewhat exhausted still we didn’t stop. We roamed around the city, ate food, interacted with each other and found more about everyone’s different & interesting projects. With this training week, the bond among us has become stronger, in fact, we are already planning to meet again very soon.

Training Week at CINECA was Great Fun!

We earned a significant knowledge about MPI, OpenMP, GPU Programming, 3D Visualisation, Blender and much more in a very short span of time. We had a tour of the CINECA Supercomputer infrastructure where we saw different (MARCONI, GALILEO, D.A.V.I.D.E.) supercomputers, how they are cooled by air & liquid, and the complex advanced network required to connect them.

Alessandro Marani gave a very useful session on how to get started with Supercomputers. I got chance to hands-on experience with HPC Galileo supercomputer, created and ran my first job script using SLURM commands to compute the program parallelly.

Nicola Spallanzani gave an excellent session on MPI (message passing interface). MPI is a high-performance computing application. It is a standardized means of exchanging messages between multiple computers running a parallel program across distributed memory for writing distributed computing applications. Each MPI process can only access its local memory.

Fabio Pitari gave a great session on another HPC API – OpenMP, which has a different approach compared to MPI. It is used for writing shared memory multi-process applications.

Dr. Andrew Emerson very well explained GPU programming with CUDA. Graphics Processing Units (GPU) are now used in HPC and machine learning applications for streaming computations. Most important vendors for HPC and machine learning are NVIDIA who also developed the CUDA language.

Then Dr. Leon Kos gave a session on an important topic – Scientific Visualisation with Paraview which is an open-source multiple-platform application for interactive, scientific visualization.

Francesca Delli Ponti gave a session on a very interesting tool to manipulate 3D objects – Blender. Blender is a free and open-source 3D computer graphics software toolset used for creating animated films, visual effects, art, 3D printed models, interactive 3D applications and video games.

Training Week – Hands on Supercomputers

The training week is finished, it’s really crazy how fast time flies! The best part of the training week was meeting many different people, getting new ideas and seeing happy faces.

Many thanks to Massimiliano Guarrasi & Dr. Leon Kos for organizing this wonderful & unforgettable training week.

All the 25 participants are now working at their respective HPC centers across Europe. I will be spending the next two months in LECAD Laboratory in Ljubljana, Slovenia. Firstly, I am super excited to work in this city. Most people speak English here, so living here will be comparatively easier for me. It is small but packed compactly with cultural highs, nature, sea, and mountains. I already have a bucket list of places I need to cover in this green boutique global destination.
I have started working on an Electrical Energy Consumption Industrial project. It has started off at a good pace and me & my mentor Dr. Janez Povh are planning to increase the project complexity. The work here is done in Slovene language and sometimes I have to translate it to English. Thanks to the handy google translator! I am really enjoying working abroad and learning a foreign language.

Stay tuned for future posts (which will be more focused on the project I am currently working on).

References: PRACE Summer of HPC Training Materials

Posted on by David Izquierdo Posted in Blogs 2019, PRACE Summer of HPC 2019 — No Comments ↓

Before entering into project-related details in the next post, I wanted to briefly write about the training week and share the experience of arriving for the first time in Czech Republic, Česká republika.

Note: since I would like everyone (versed or not in the subject) to understand these posts, but also to keep it to a pretty detailed level, all the concepts that may sound confusing for a newbie will be hyperlinked to an article or a video where an explanation is given; this applies to all the posts that I will publish from now on. Now, let’s go…

The training week at CINECA was a great time from a personal perspective, as I mentioned in my previous post. But it was also very interesting in terms of what we learned. Even if I had some experience working with clusters, learning how to program in order to exploit the capabilities of the architecture of the specific computer that you are working with is a very valuable piece of knowledge. Basically, it helps you to treat a computer from a different mindset, taking more into account the hardware that is actually available to you. This is especially true for the case of someone like me, who is an aerospace engineer with limited programming knowledge in low-level languages (although a programmer would not agree with me in the definition of low-level language, for sure…).

The journey towards Ostrava was enjoyable. I spent almost the whole flight from Bologna to Praha (no exonyms here) talking to a very nice Czech guy who seemed to know everything about his country. All the fears of coming to an unknown country were gone by the end of the flight.

Pendolino train (Praha – Ostrava). Image by Martin Hefner

The next stage was getting the train towards Ostrava. The ticket to cross the country from West to East ends costed less than a meal. Which is particularly surprising if one takes into account that meals are here way cheaper than in Western Europe. The blue train connecting Praha and Ostrava (among other cities), called the Pendolino, is not extremely fast but is great to get an overview of the country.

Once in Ostrava, it was easy to realize that many people do not speak English; not a single word. However, this is not a great deal since they are generally very welcoming and try to help you anyhow.

The arrival at the accommodation site was smooth and the hotel happens to be just a few meters away from the supercomputing centre. By the way, the IT4Innovations National Supercomputing Centre is a brand-new almost futuristic building, a very confortable place to work in.

IT4Innovations installations. Image by David Izquierdo.

Setting aside the aesthetics, the supercomputer center owns several HPC clusters with different characteristics. The main ones are Anselm (3344 cores, over 94 TFLOP/s theoretical peak performance) and Salomon (24192 cores, over 2 PFLOP/s theoretical peak performance ). Next year they are planning to add a new petascale one, which will rank among the best ones in both Europe and the world.

In a few days time, I will publish a post talking about the details of the project and the first steps, but I thought it was first important to write about the context in which it will be developed. I hope everyone enjoyed it.

Posted on by Igor Kunjavskij Posted in Participants 2019, PRACE Summer of HPC 2019 — No Comments ↓

Hi everyone, I’m Igor and am quite thrilled to be participating in this years Summer of HPC program organized by PRACE! Currently I am in the last term of my master studies in the field of Engineering Cybernetics at the University of Stuttgart in Germany. After initially specializing during my bachelor studies in robotics and control engineering, I slowly started to deviate towards the field of Artificial Intelligence during my master studies. There I “inevitably” ran into the field of High Performance Computing (HPC), as it’s impossible to say Deep Learning without asking for computing resources in the same sentence. This led me to spending most of my free time during my master studies at the High Performance Computing Center in Stuttgart, which is conveniently located just next to my University. Here I was introduced into the exiting world of HPC and all of its intricacies. Here I also heard for the first time about this program and was immediately hooked by the idea to participate in it. One year and an application procedure later I am now happy to be one of the selected participants of this year’s Summer of HPC program!

Now during the first week of the program hosted by CINECA, the High Performance Computing Center located in Bologna, I got a little refresher on most of the important topics in HPC. These ranged from parallel computing with MPI and OpenMP to using CUDA in case a Nvidia GPU is available. We also had a crash course on how to how to visualize our results and to reach out with these on social media.

Some of the compute racks in CINECA cooled in an extra chamber

Our hosts at CINECA gave us a tour of the Supercomputer, providing us with interesting insights and the chance to look into the “heart” of a High Performance Computing Center. Next to that we managed to get a lot of impressions of the culture in Bologna and had a short tour of the city center.

CINECA also had a lot of historical exhibits available, here a 4 KB block of RAM from the 60s. An iPhone X has nowadays almost a million times more than that in it.

After this week I will be heading up north towards Ireland to stay and work in Dublin on my project at the Irish Centre for High-End Computing (ICHEC). Here I will delve into the topic of Deep Learning on the edge, so stay tuned for more!

Posted on by Ebru Diler Posted in Participants 2019, PRACE Summer of HPC 2019 — No Comments ↓

WHO AM I ?

‘‘ Hello world !’‘

I am Ebru, I completed my bachelor degree in Dokuz Eylul University as Industrial Engineer in Turkey and I am doing my master degree in same department.

I got introduced to software by my interest in algorithm competitions. I was one of the representatives of IEEE Student Branch in my own school. I heard about the IEEEXtreme contest. In 2017, I got zero points from the competition. Wheres after that I practiced my algorithm skills on Hackerrank. In 2018 IEEEXtreme, my team got 5th place in Turkey among 100+ Turkish teams.

I work hard on optimization problems as an industrial engineer. Some optimization problems can be solved by linear programming. The solution of these programs requires a very long time. For example: travel salesman problem solution lasted 41 hours on my computer. This inefficiency was very contradictory to the vision of engineering. Then I noticed the existence of HPC. Perhaps for the first time in my life I would see a high performance computer with this project!

I like racing against time. I received degrees in some competitions held in Turkey. Here are couple of the hackathons that I’ve won: Hack4energy , Hugoboss Hackathon, Anadolu Sigorta Hackathon, Yapıkredi Hackathon, Code.Masters , Mercedes-Benz Hackathon etc.


My current challenge is Summer of HPC. Now I’m trying to understand the HPC in a language that I don’t know, in a country I have never been in before.

I’m very happy to be at the EPCC in this new challenge. As you know :

”If you can dream it, you can do it.”
Walt Disney

To be continued…

Posted on by Allison Walker Posted in Blogs 2019 — No Comments ↓

Buon giorno! I am currently sitting on a plane headed for Amsterdam where I will begin my HPC project, and I thought I would take this opportunity to compose my next blog update. 

We have officially finished our orientation week in Bologna! This past week was full of so much learning. With classes from 9 to 6, Monday to Friday, we managed to cover a lot of content: principles of HPC, Parallel Processing, OpenMP, MPI syntax in C (note: I am not a C programmer), CUDA, remote visualization with Paraview, Blender, CINECA’s Galileo and Marconi clusters, and more…

Hanging out with Marconi

All of the above were essentially foreign to me before this week. The SoHPC coordinators have taken great effort to ease even the least computer science-y of us into the realm of HPC in an engaging way. 

My favorite part of this past week has been getting to know my fellow interns. We have since dispersed to different corners of Europe: eleven HPC sites across eleven different European countries host the students for the remaining of the summer, where we are each allocated to our own unique project. But for this first week, we all had the opportunity to get to know one other, help each other out with classwork, enjoy the Italian heat, and bond over (too much) pizza, pasta and gelato! 

The SoHPC 2019 crew

If this first week was any indication of what’s to come, I’m up for a fantastic Summer of HPC! 

Stay tuned for my next update, coming at you from Amsterdam 🙂

Posted on by Martin Molan Posted in Participants 2019 — No Comments ↓

My name is Martin Molan. I come from Slovenia and I have just finished the first year of master’s program at IPS at Jozef Stefan Institute in Ljubljana. Before that I obtained a BA in mathematics from University of Ljubljana.

In front of supper computer at CINECA

I applied to summer of HPC – and to the anomaly detection project in particular – because I am interested in applied machine learning on big datasets. Besides studying I work on machine learning systems for responsible gambling (identification of risk for gambling addiction from user data).

Alongside other judges (I am in the middle) at Winter holidays open debate tournament.

In my free time I like debating at university debate tournaments (if fact if I weren’t accepted to summer of HPC I already planned to go to debate camp in Porto) and judging and coaching high school debates.

Posted on by Dimitrios Voulgaris Posted in Participants 2019 — No Comments ↓
Froma a previous summer school at IHP, in Frankfurt am Oder.

Everything started so unexpectedly, and yet here I am in a beautiful country with some amazing people learning about the worldwide scientific shift. But let’s go to the beginning of this adventure.

First things first, I am Dimitris, 23 years of age and I think it was a February night when I first found out about the PRACE idea. Being in my 10th, and last, semester of my studies in Electrical and Computer Engineering in Greece with focus in parallel computer architecture, I have quite an experience in summer schools and internships; yet this particular one seemed different. Supplying with more than a satisfying number of projects in seemingly diverse, yet computer – centered topics, PRACE SoHPC (as it is called), seemed engaged in finding a perfect match between one’s (i.e. our) individualized interests and the scientific requests (at least it worked for me, allowing my experimentation in “Heterogeneous Memory Data Placement“) while promoting the cultural exchange.

“The past few days brimmed over fun and amusing conversations somehow perfectly combined with captivating lectures on interesting topics”

During our stay in Bologna we had the pleasure to be hosted by CINECA and visit their supercomputing site

Keeping in mind that one of my biggest “addictions” is exploring the world while meeting fascinating peolple (gym and books should be added as well), I would like to emphasize on this aforementioned cultural interraction. Called to split my summer between Bologna and Barcelona (at BSC), I knew that various and unique experiences were implied. Indeed, the past few days brimmed over fun and amusing conversations somehow perfectly combined with captivating lectures on interesting topics therefore not only living up to my expectations but rather exceeding them.

Trying to sum up (I will provide a more intriguing post about the Bologna experience), I would like to state my thought: I am not sure if it was me that selected PRACE SoHPC or if PRACE SoHPC selected me, nevertheless I am confident that a once-in-a-lifetime learning and exploratory experience has just begun and seems promising.

Should you be interested in keeping up with me in this unique journey, do visit my page for future posts and also check the following link:

http://www.linkedin.com/in/dimitris-voulgaris-dv1

Should you want to take an overall view of the program, check out the latest posts of my colleagues:

Posted on by Noë Brucy-Ciaramella Posted in Participants 2019 — No Comments ↓

About me

Hi, I’m Noe

I’m a French student and I do astrophysics, studying disks in the Universe.

In short, I’m making numerical simulation of the gas orbiting young stars, trying to investigate how planets form. I made a short video, where we see it rotating. Have a look a it !

 

https://peertube.fr/videos/watch/80fc38ad-3593-49e5-ac15-a3ebae8ca355
Video of a simulated protoplanetary disk.

My work is not just making nice videos, I also spend a lot of time writing code, solving bugs and trying to understand  the physics explaining what is happening my simulations.

Next year I will do a PhD and I will try to run more ambitious simulations of a whole galaxy, trying to get the best resolution possible. That is still a disk but at a (much) bigger scale! I will use the next generations of supercomputers, and take advantage of the upcoming GPUs architecture. That means that I need learn how to compute on graphic cards instead of processors, and it’s why I applied to SoHPC.

M51
The Whirlpool galaxy. I made this image with friends at the Observatory of Haute Provence.

My SoHPC Project

The goal of my SoHPC project is to compute interaction between molecules, and to do it fast using GPUs. That may sounds far for astrophysics but the electrical force between two molecules is very analogous to the gravitational force between stars in the Galaxy. So I have just to replace “molecule” by “star”, “charge” by “mass” and I can apply what I will learn this summer to my favorite field.

The training week

The first week in Bologna is all about meeting new friends, mixing up languages between English, French and Italian, eating Pasta and Pizza and learning how to do parallel programs. We are a lot, and each one of us has a different culture that is worth sharing. A fair amount of the day is spent in training lessons, but we still have plenty of time to exchange and appreciate the stay together.

The survival kit in Bologna. Yes there is sun, and it is really hot.

What’s next ?

On Saturday I will take the train to Jüelich in Germany (that’s almost 13 hours because I chose to take the night train), the first step of a new adventure. In the next post, I will talk about GPUs, tasks, and I hope, bikes …

Posted on by Khyati Sethia Posted in Participants 2019 — No Comments ↓

Hi there, I am super excited to welcome you!

My name is Khyati and I come from Jaipur (India) – The Pink City. I am studying a Master’s in Data Science at the University of Salford. I am always passionate about continuous learning, finding new and exciting technologies which can prove useful for my data projects. I have worked on Data mining (Prediction/Forecasting), Big Data, Advanced Databases, and Text Mining data problems which involved processing, manipulation, statistical analysis, visualization, and modeling (the link to resume is inserted at bottom of post for more details).

But when I worked on real-world big data & deep learning problems, the computation of application would take forever. This was when I considered applying to the PRACE Summer of HPC program, where few projects were exactly what I needed. I am very ecstatic that I was chosen among top applicants from all across Europe and could work on a problem of my interest in the leading supercomputing centers in Europe. The fusion of Supercomputing and Data Science enables me to perform intensive processing on limitless data. I will be spending my summer in LECAD Laboratory, Ljubljana, Slovenia. I did my bachelors in Electronics & Communication field; hence I have chosen to work on Industrial energy consumption big data. In this project, I will create & store Big Data (BD) files, prepare them for analysis, build and evaluate the predictive model using RHadoop. By using HPC & Big Data, I will like to educate the Energy Community. I will also prepare materials for MOOC entitled MANAGING BIG DATA WITH R AND HADOOP which will be made available online. Parallelly, I will be working on another project related to Human activity recognition in smart houses.

Apart from this, I like to explore new places and volunteer in different types of events. I also enjoy dancing and cooking for fun.

Exploring the historic city - Bologna, Italy!
Exploring the historic city – Bologna, Italy!

My time is up, and I thank you for yours! I’ll be back again with details of my project as I progress, so stay tuned and visit again for exciting future posts.

Feel free to contact me for any question related to SoHPC or the project I will be working on.

Contact: +44-7739897649 | LinkedIn | Resume

Posted on by Antonio Miranda Posted in Participants 2019 — No Comments ↓

Hi everyone. I am Antonio Miranda, writing from Bologna, where the second step of this adventure is taking place!

And I say second because, as opposed to many of my colleagues, my journey actually started in Barcelona. There I had the opportunity to visit BSC, meet leading-edge researchers working on HPC and see one of the largest supercomputers in the world, Mare Nostrum: 

Fun fact, BSC supercomputer was built inside a church.

I graduated from Biomedical Engineering and right now I am studying a master on Big Data at University Carlos III, in Madrid (Spain). Despite I specialized in Deep Learning during my master, this summer I will travel to GRNET, in Athens, to complete a project on drug discovery using Free Energy Perturbations.

In addition to solving problems in life sciences, I deeply enjoy talking with friends about politics, history of science, the Marvel Universe…anything with a beer in our hands, plenty of time and without any worry.

In the following weeks I will be posting my work, my thoughts and adventures in Athens. Keep an eye on my posts to walk this path with me!

Posted on by Arsenios Chatzigeorgiou Posted in Participants 2019 — No Comments ↓

What does a biologist do inside a supercomputer facility?

A biologist next to an HPC (Marconi @ Cineca).

In order to answer the question, I have to introduce myself first.

Hello, I am Arsenios Chatzigeorgiou, I am 27 years old and I have graduated from the Biology Department in Athens, Greece.

During my high school years, I was very fascinated with almost every science field that came to my view. Maths, physics, chemistry and biology, but at one point I had to choose a field to study.

During my graduate degrees in biology, in a course about bioinformatics, I got really excited writing my first scripts. I remember it felt like a child playing in the playground. It was therefore a no brainer choice that I would continue my studies in a more informatics field, enter, Master’s degree in Biomedical Data Science. In all those years however, I was very excited to learn stuff about other sciences.

A biologist in a biochemical lab, a few years back.

So, we reached at the point where during my Master studies, I first learned about the PRACE program, and the use of HPC in science and research. I learned that HPC has made it possible to simulate and visualize complicated biological and physical processes. When I heard the great opportunity the PRACE program offers to young scientists, with the Summer of HPC, I was determined to do my best to be a part of that. The fact that there was a project that offered me the opportunity to work on a more physics related subject, was also a big plus. So, around April, when I was informed that I was selected for a project about visualization I was extremely happy and hyped about the exciting summer.

Indeed, the summer came and this week, from the first of July, I am in Bologna, Italy along with all the SoHPC participants, getting the necessary training for the use of HPC. Next week I am going to be in Slovenia, in the University of Ljubljana at the LECAD (Laboratory for Engineering Design) team. There, I’ll try to visualize some gyrokinetic data using an open source Python library.

A Bologna tunnel leading to knowledge.

Pardon my long story, but get ready because next week I am going to describe the learning outcome and the awesome training experience, here in Bologna, Italy.

Stay tuned!

Posted on by Caelen Feller Posted in Participants 2019 — No Comments ↓

About Me

Hello! My name is Caelen, and I’m from Ireland. I’m currently in the final year of my undergraduate degree in mathematics in Trinity College Dublin. I love interesting solving interesting problems, and sharing them with others. When I started programming, I was amazed by how useful mathematics was, and the way in which it lets you create solutions not possible otherwise – as a simple example, what’s fastest way to add the fibonacci numbers together (these are 1, 1, 2, 3, 5, 8,…)? This is the main reason why I am study mathematics, and why I’m always excited to apply what I’ve learnt to programming.

Paper crane, rendered on Galileo in 1 minute.

The image above is a paper crane, a traditional origami model. I created this image in Blender myself, rendering it on the Galileo supercomputing cluster at CINECA this week! My other hobbies include games of all types and their design, particularly board games, reading, knitting and hiking. Some favorites include “Terra Mystica” and “Splendor” among games, and “Children of Time” and “Annihilation” among books. I’m sure I’ll slip in some more about these things at the edges of future blog posts, watch this space!

  • Orange
  • RGB
  • Blue

Training Week @ CINECA, Bologna

This week, I am staying in Bologna near the CINECA high performance computing (HPC) research center. The other participants and I have been given a thorough crash-course in the principles of HPC and a refresher on the best principles of social media. While I’ll do deeper dives into these topics later and how they apply to my project, I’ll give a quick summary now!

The first focus has been social media – we’ve been taught the best principles of blogging and use of micro-blogging platforms such as Twitter. For example, do you know how search engines find your posts? They don’t normally take the time to review each word you say, so proper use of social media tags are essential. I’ve tagged this post under origami, which means that both users of the site and search engines will be more likely to associate it with the correct subject matter. Use of too many tags will upset search engines however, triggering spam filters, so be careful!

We’ve also been taught the basics of parallel computing on a large cluster like Galileo here at CINECA. Running your code on a parallel computer boils down to running many, very similar programs at once, all of which can communicate (ideally without waiting for the others). If we want to run all of these programs across different computers in a supercomputer, we use MPI (Message Passing Interface), a programming library which allows you to communicate over a network, like the wires shown below in the MARCONI supercomputer, the largest cluster in CINECA. This communication is quite slow, so we try to keep programs as separate as possible.

Nework wiring necessary to run MARCONI

If you only want to run your program on all of the different processors of one computer, we use OpenMP (Open Multi-Processing). Here, we can communicate quickly and efficiently, so we can adopt a different style when programming, which allows us to say keep a running total of results from all the processors without slowing their computation down unnecessarily. This is what is used inside of Blender to render images like those above in parallel.

While there are many tricks to handle more complicated situations, making a problem run in parallel works best when it can be broken down into small independent blocks. For example, when rendering the images above, I instructed Blender to break down the image into small squares. It then gave the squares to the 16 processors I had been allocated on Galileo, and figured out what the image would look like using the Cycles render engine and the models I gave it.

My Project @ EPCC, Edinburgh

I’ll be working in Edinburgh at EPCC this summer, on the project “Parallel Computing Demonstrators on Wee ARCHIE”. Wee ARCHIE is an unusual computer – it is in fact made of 16 Raspberry Pi chips – each very small, lightweight computers connected together so that they can work as a single machine. To each chip is attached an LED panel, allowing me to display what is occurring on the chip and how it is communicating to other chips. I will be implementing a coastline management program – simulation of flood and wave barriers, and using the LEDs to make sure it’s very obvious how this is happening in parallel to a non-expert in computing.

Picture of Wee ARCHIE
The Wee ARCHIE computer (from the project description page)

On the weekends in Dublin, I teach extracurricular classes in maths to secondary/high-school students in a local university, but I’ve never gotten a chance to explore computer science education. I’m excited to work on such a visual, interactive project! I’ll fill you all in on the project in more detail when I hit the ground in Edinburgh.

Check out my website for more about me!
You can find more information about my project in its official description.

Posted on by Davide Di Giusto Posted in Participants 2019 — No Comments ↓

Hi there, summer of High-performance computing has finally kicked off at CINECA, in Bologna, with the training week. Despite the high temperatures, we have been working quite hard, but hopefully, it will pay-off later.

Here you can see me chilling a little bit between sessions, just outside the supercomputer room.

Useless to mention, everybody is super excited to reach the SoHPC sites next week. For what concerns me, I will head to the STFC Hartree centre, in Warrington, a village just outside Liverpool.

My enthusiasm for SoHPC as perfectly embodied by Liverpool Manager Jurgen Klopp.

But let us back up a little bit: my name is Davide and I come from Udine, Italy where I completed the MS in mechanical engineering. As final year project, I performed intensive DEM simulations of granular flows, which lead me to develop a keen interest in HPC and complex fluids.

About me, as you might suspect, I am a football fan, but also have other interests like British rock music and trips into the nature.

Then, stay in touch if you want to hear more about my project and other interests while I explore the UK. In the meantime, feel free to join the discussion by commenting on my posts.

See you!

Facebook
Twitter
Follow by Email