Avalon Servers

The Avalons are our 4 servers that we use to run many processes in parallel. Have a look at Nils’ talk for the basics.

Configure SSH Hosts and use SSH Keys

This makes the whole usage of the Avalons (or any other remote Unix computer) much more comfortable. With SSH keys, you don’t need to type your password anymore, when you want to log in or copy to/from a remote computer. (It is also considered much safer to use this for log in whenever you can, best practice would be to allow log in to avalons only via this, but we also allow log in with password for now.)

If you don’t have a SSH key yet (check if the folder ~/.ssh exists), create one with:

ssh-keygen

It should be fine to use the recommended settings. This creates the folder ~/.ssh. There should be a public (id_rsa.pub) and a private key file (id_rsa) in that folder. If you want to use SSH in other situations (e.g. for gitlab), make sure you never copy the private key by accident. Never give the private key to anyone. In my folder there are some additional files:

/home/ilyas/.ssh
├── config
├── id_rsa
├── id_rsa.pub
└── known_hosts

Create the file ~/.ssh/config and open it in an editor. There you can define what the IP addresses of hosts are, with which username you want to log in, etc.. For the avalons, put the following into the file:

Host avalon
     HostName XXX.XXX.XXX.XXX # ask colleagues for IP
     User ilyas
     Port XXXX # ask colleagues for port

Host avalon01
     User ilyas
     ProxyJump avalon

Host avalon02
     User ilyas
     ProxyJump avalon

Host avalon03
     User ilyas
     ProxyJump avalon

Host avalon04
     User ilyas
     ProxyJump avalon

Replace my username ilyas with your username(s) on the avalons. The first entry avalon is actually the head node, not one of the actual computing servers, I hope that is not too confusing …

Now in a first step, try to log in to the head node, by entering:

ssh avalon

This should give you some fingerprint that you can accept. After that it will ask you for your password. Make sure it works and you are logged in. Then log off again by pressing ctrl + d. Now copy your ssh key to the head node with:

ssh-copy-id avalon

For that you have to enter your password as well. From now on, you should be able to log in without a password, but rather with your SSH key. Try to log in again with

ssh avalon

Once that works, you can to the same with one of the avalons. E.g. if you want to use avalon04:

# make sure your password and account are set up by logging in once:
ssh avalon04
# log off again, then enter:
ssh-copy-id avalon04

That’s it!

rsync instead of scp

In many situations, rsync has some advantages over scp. It is quite easy to use rsync to copy only files that are newer in the source directory than in the destination directory using the option -u. For that to work more neatly when copying files back and forth between different machines/drives multiple times, I recommend to also use the option -t to keep the time stamps when you copy files.

I simply use rsync with the following options most of the time:

rsync -ruvt src dst

The options -ruvt include the two options described above. In addition, -r makes it recursive (includes sub-folders), -v (verbose) leads to some more feedback printed to the command line.

Example

TODO

Let’s say you want to copy your local folder …