SSH – Secure Shell access to HPC systems

To use the HPC systems at RRZE, you have to log into a cluster frontend via an SSH (SecureShell) client. SSH is a common command-line tool for remotely logging into and executing commands on a different computer over the network. The following topics will be discussed in detail on this page:

Basic usage

Connect to a remote host

Under Linux, Mac and recent Windows 10 versions, a command-line SSH client is pre-installed. If you want to have a graphical user interface, you can use third-party clients like PuTTY (Windows, Linux) or MobaXterm (Windows).

Direct access to the cluster frontends is restricted to networks within the university. So if you are connected via such a network, or if you are using VPN, you can connect using the following command:

ssh USERNAME@CLUSTERNAME.rrze.fau.de

In this case, USERNAME is your HPC user name and CLUSTERNAME is the name of the cluster you want to log into, e.g. woody, emmy or meggie. If you want to access TinyFat, TinyGPU or TinyEth, you also have to connect to woody. You will be prompted for your HPC password or your SSH key passphrase if you are using SSH keys. After successful authentication, you have a login shell on the target system.

If you are outside of the university network and are not using VPN, you have to connect to the dialogserver first :

ssh USERNAME@cshpc.rrze.fau.de

You can then use the above SSH command to connect to the cluster front ends from there.

Copy data to a remote host

A secure mechanism for copying data to a remote host is also available in all OpenSSH distributions on Linux, Mac, and current Windows 10 versions. When running Windows, you can also use WinSCP, which has a graphical user interface.

For all command-line based options, the secure copy mechanism is invoked by the following command:

scp <filename> USERNAME@CLUSTERNAME.rrze.fau.de:<remote_directory>

This will copy the local file <filename> to the directory $HOME/<remote_directory> on the remote system. This directory must exist prior to the copy attempt. Keep in mind that nearly all available file systems are mounted on all frontends (see File Systems documentation). It is therefore sufficient to copy data to only one frontend, e.g. cshpc.

Graphical applications

We generally do not recommend to run graphical applications on the cluster frontends, since they normally consume much more resources and can, therefore, interfere with the work of other users on these shared systems. However, there are some cases where the use of graphical applications is necessary.

For applications that do not need many resources, it should be sufficient to enable X11 forwarding/X11 tunneling by your SSH client via the -X option:

ssh -X USERNAME@CLUSTERNAME.rrze.fau.de

However, this requires an X11-Server running on your local machine, which is generally not available by default on Mac and Windows. In this case, you need to activate X11 tunneling in your client configuration, as well as have an X Window server (e.g. Xming or MobaXTerm for Windows, XQuartz for Mac) running locally.

As an alternative, we recommend using remote desktop software to run graphical applications, e.g. NoMachine NX. A description of how to set up and use NoMachine NX on cshpc is available in the dialogserver description.

SSH public-key authentication

As an alternative to logging in with your HPC password when you connect to a server via SSH, you can also use public key authentication. It requires a so-called SSH key pair comprised of two matching parts – a public and a private key. The key pair is generated on your local machine. The public key is uploaded to the remote system, whereas the private key remains on your local machine. We recommend generating a separate SSH key pair for every system (workstation, laptop, …) you use for logging into the HPC clusters.

Generating key pairs is possible when your client has OpenSSH capabilities (Linux, Mac, Windows 10). If you are using PuTTY, you can generate keys with puttygen.exe.

When generating a key pair, you have to choose between different algorithms and key sizes. The recommendations which one to use are changing over time since also the capabilities to break encryptions increase. Currently, it is advised to use either rsa with a length of 4096 bits, ecdsa with 521 bits or ed25519. Use one of the following commands to generate a key pair:

ssh-keygen -t rsa -b 4096

ssh-keygen -t ecdsa -b 521

ssh-keygen -t ed25519

During the generation process, you will be prompted for a passphrase to encrypt your private key. We don’t recommend leaving this empty since in this case, your private key sitting on your computer as a plain text file.  If this unencrypted private key is copied/stolen by someone, they can access the corresponding server directly. In case it is encrypted by a passphrase, the attacker must first find out the passphrase in order to gain access to the server with the key.

By default, the key pair is generated into the folder .ssh in your home directory, with the files id_<algorithm> being your private and id_<algorithm>.pub being your public key. If you want to change the location and name of your key pair, use the following option:

ssh-keygen -f <path_to_keys>/<keyname> -t <algorithm>

The public key must then be copied to the server and added to the authorized_keys file to be used for authentication. This can be conveniently done using the ssh-copy-id tool:

ssh-copy-id -i ~/.ssh/id_<algorithm>.pub USERNAME@cshpc.rrze.fau.de

If this doesn’t work, you can also manually copy the public key and add it to ~/.ssh/authorized_keys:

cat id_rsa.pub | ssh USERNAME@cshpc.rrze.fau.de 'cat>> ~/.ssh/authorized_keys'

Once the public key has been configured on the server, the server will allow any connecting user that owns the private key to log in. Since your home directory is shared on all HPC systems at RRZE, it is sufficient to copy the key to only one system, e.g. cshpc. It will be automatically available on all others.

If you have changed the default name of your key pair, you have to explicitly specify that this key should be used for connecting to a specific host. This is possible by using the -i parameter:

ssh -i ~/<path_to_keys>/<keyname> USERNAME@CLUSTERNAME.rrze.fau.de

For frequent usage, this is quite cumbersome. Therefore, it is possible to specify these parameters (and many more) in the ~/.ssh/config file. A detailed description of how to do this is given below.

If you have problems using your key, e.g. when you are asked for your password despite the key, or in case authentication is not working for some other reason, try using the option ssh -v. This will cause SSH to print debugging messages about its progress, which can help locate the issue much easier.

SSH agent

If you have set a passphrase for your private SSH key, you will be prompted to enter the passphrase every time you use the key to connect to a remote host. To avoid this, you can use an SSH agent. After you have entered your passphrase for the first time, this small tool will store your private key for the duration of your session. This will allow you to connect to a remote host without re-entering your passphrase every time.

If you are using a current Linux distribution with a graphical desktop session (Unity, GNOME,…), an SSH agent will be started automatically in the background. Your private keys will be stored automatically and used when connecting to a remote host.

In case you are not using a graphical desktop session or your SSH agent does not start automatically, you will have to start it manually by typing the following into your local terminal session:

eval "$(ssh-agent -s)"

This will start the agent in the background. To add your private key to the agent, type the following:

ssh-add ~/.ssh/<keyname>

After you have successfully entered your passphrase, you will get a confirmation message that your identity file was successfully added to the agent. This will allow you to use your key to sign in without having to enter the passphrase again in the current terminal session.

You can also list the keys which are currently managed by the SSH agent via:
ssh-add -l

For more information about the SSH agent, type man ssh-add on your terminal.

Configure host settings in ~/.ssh/config

If you are regularly connecting to multiple remote systems over SSH, you’ll find that typing all the remote hostnames, different usernames, identity files, and various more options is quite cumbersome. However, there is a much simpler solution to define shortcuts for different hosts and store SSH settings for each remote machine you connect to.

The client-side configuration file is named config and is located in the .ssh folder in your home directory. If it does not exist, you can create it manually.

The configuration file is organized in different sections for each host. You can use wildcards to match more than one host. The SSH client reads the configuration file line by line, so later matches can override earlier ones. Because of this, you should put your most general matches at the top of the file.

One simple example to create a shortcut for connection to cshpc is given below. The following is added to ~/.ssh/config:

Host cshpc
  HostName cshpc.rrze.fau.de
  User USERNAME
  IdentitiyFile ~/.ssh/private_ssh_key_name

With this configuration, you can now connect via

ssh cshpc

instead of typing

ssh -i ~/.ssh/private_ssh_key_nameUSERNAME@cshpc.rrze.fau.de

A large number of different SSH options are available. Some options which are used more frequently or are especially useful are listed below. You can find a full list by typing man ssh_config in your terminal.

Security recommendations

In general, it is recommended not to trust systems that are accessible to multiple users or that someone else has root access to, which is true for all HPC systems. Even with best efforts by the administrators to keep the systems safe, it is always possible that attackers are able to gain root rights on the system, which makes them very powerful. An attacker may for example install keyloggers or hijack your running SSH-agent, just to name a few possibilities.

Thus it is often recommended

  • not to log in via interactive passwords on untrusted hosts,
  • not to use SSH agents on untrusted hosts,
  • and not to use SSH agent forwarding to untrusted hosts.

It is generally more secure to use SSH public-private key pairs for authentication when accessing remote systems, as long as these rules are followed:

  • Store no private keys on untrusted hosts. Private keys should only be placed on single-user systems (e.g. your laptop).
  • Always use SSH private keys with strong passphrases.
  • Use only one SSH key pair per system with shared homes.
  • Use a separate key pair for every client (laptop, desktop,..).

To make it easier to jump between different systems at RRZE, we recommend generating a separate key for internal use only. This key may also be used for access to external systems (e.g. LRZ).

SSH agent forwarding

SSH agent forwarding is mostly used as a Single-Sign-On solution to connect from one remote host to another (e.g. from cshpc to other cluster frontend or between different cluster frontends). When you enable SSH agent forwarding, the query of the remote server for the private key is redirected to your local client where the SSH-agent is running. This eliminates the need for using password logins and for having private keys on remote machines. However, it is not recommended to use SSH agent forwarding to an untrusted host. Attackers with the ability to bypass file permissions on the remote machine can gain access to the agent on your local machine through the forwarded connection. An attacker cannot obtain key material from the agent, however, they can use the loaded keys to gain access to remote machines with your identity. An alternative to using SSH-agent forwarding is the ProxyJump functionality provided by SSH, which is described below.

X11 forwarding

Similar to SSH agent forwarding, X11 forwarding can be a security risk. If your SSH client is configured to generally allow applications on a remote server to render GUI windows on your screen, this can be exploited by an attacker. It is therefore recommended to specify ForwardX11 no for all hosts in ~/.ssh/config and only use -X on the command line when necessary.

Host keys

SSH host keys are used to verify a server’s identity before you sent any sensitive information like passwords to it. Each server has a unique host key, which is the server’s public key. It can be used by the client to decrypt an authentication message sent from the server when connecting. This makes sure that the remote host you connect to is really the one you intended to connect to, and that your connection is not secretly redirected to another server.

SSH clients automatically store host keys for all hosts they have connected to. These keys are normally stored in ~/.ssh/known_hosts. If the host key of a server you are trying to connect to has changed, you will get a warning message.

When you connect to a server for the first time, you cannot know if the key offered by the server is correct. Therefore, we provide the public system keys for the cluster frontends below, which can be directly added into the ~/.ssh/known_hosts file (you may need to generate the .ssh directory and/or the file if it does not exist yet) on your local machine.

cshpc.rrze.fau.de ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAs0wFVn1PN3DGcUtd/JHsa6s1DFOAu+Djc1ARQklFSYmxdx5GNQMvS2+SZFFa5Rcw+foAP9Ks46hWLo9mOjTV9AwJdOcSu/YWAhh+TUOLMNowpAEKj1i7L1Iz9M1yrUQsXcqDscwepB9TSSO0pSJAyrbuGMY7cK8m6//2mf7WSxc=
cshpc.rrze.fau.de ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPSIFF3lv2wTa2IQqmLZs+5Onz1DEug8krSrWM3aCDRU
cshpc.rrze.fau.de ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNVzp97t3CxlHtUiJ5ULqc/KLLH+Zw85RhmyZqCGXwxBroT+iK1Quo1jmG6kCgjeIMit9xQAHWjS/rxrlI10GIw= 

emmy.rrze.fau.de ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2q7Ung+RdwLkMyQXiod/6BFsUBMcKlEnvG3pFR7cw7/wdLUcjUU4ubQR9ctNlQZok7XU9b2ttMVwUOYI3w2RZnQFwm9jzUbAAl00XRfBThI9cWlgJu0UR/I+W/iRJdBSAmffwsQYTYBzJ4cRTtKSLZ98yEbJVtwfRRG12PVMewNGVDsnmBOBX5zWG92tgaA1bXAiB0GVWBS79lV78+ii/1UR/PldZaA+RQtxDx0ckuc8vq10XK4GvXJijyrEzIsi3SeIFApMhr+W84uIGp5HjhaaYwVWMkBge8PX8bR8oXNaUFLVmaRUrX/WSchCmLp2YBh3npeZ/B9vAtb6LXoS7Q==
emmy.rrze.fau.de ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBqBH0GRzrNUrTyOE25TkQXqY/30PLVUqUam93XArPMb
emmy.rrze.fau.de ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEsYP0xDLyI/VHC68o4BqZ1RR5Ff7qMscZjKiKD1kEP2ckea0dMdH4oB4ahScShcEG5iZmQ2FlN41FbGX4zp6Go=

woody.rrze.fau.de ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDeiwzMm9JQ3fnc7AqLkXrlmUOHh5CXh11XF0gQe0wqZx2sER+oumfi+T0M2lWMTBoeMOL74fMal/Bgpq66ETqnodDyVasyD6LwaJwEVIxlss9gcrN1SPa1XaXxgAhpEaR7mTwrNLjM6W5d3+6CMiLvp32lsL4RrQHShjhkXAYrhr3RbApMLwFdb6QZzHa7teN47aMy9s6oubRr4haoeTbfFGRaQjIyguG18nrOcnTlhyPafiHyivL5AE0wLiLCZyqux5Q0GZhIr1uK8smyT1QIIf55A8bRHVvE/QGkcT8lz2w9OnKJKNHUS/UA4MerJIM5V2/IOOdSeDgLnMuJ0usEwawgqNXoqx76X1wuhXA8IqaP4J0vo2OkK7QdyZP7qKP2YMZoDwFmyl73C8xaCw28ovIYGzPmCVLpwtIAQf0uX5xe2yWo9hLPhfP3rTKEKksOVnLKcLNpMLMqxxJHsHbLnmbFn7RdDWQ92JH0nZDAjmUZ2NaHzoPbcz5y1/CCvdURUrNLSosaMqcclq5yZif8uWtUQ8wvIacrMQUFetPTz86dg9ryIrZhxOaYgWzNQVV2ZeED4k5P+QQLkjbvw26htYWxHP7BpTOxIYryQJO/gRMTOnDPP/js49nECn1bW07HDYznhhztGVcjZAgNND8hELHxAmG3WYAsR0/sOMM1ddddM3GbYaCzX++3EE26dvEWpy3J6rHRq9mvGhRG7p8Y2LozlyDXo8wodNNci2/kXTArgeZnU5W15awjl9G5haPcoeNxg467T7bIKGq9JdLkHhqBrqGesrM0ADcDLufgrcT0SIukrc9rSOgVWtYfnXeRWfj7FrjaT15FpWeFSxBXqkQeOrScrPpmbkE7fR5xJYPFDugXQs2FvIjfvW8TsSWaxyt7eLbiFdfa1czGO1S5SOIASIn4/6CuECasvMalSX0JKKLV3Yhs7zXMk3t3wiAHXJ8m+PZB7sY0jhU1UDJIymbvwSzEtrRpbXLkQDhf9XHuG0yNS8dd9u25X6jqoWogPGKoEpQX/2xicebMfJRA3TLWuOM4RtqwAYNVrjsmAfXVmAewvlAtPNFrD0JeKJANVGfa6JFvLfhGHI2UVmdt5vzQzneI/31/+2jNbglcheAfsUO5gPbq3BdToM1bDxJ8hWw3sS2gZ2DZybVz/95rdh9zcj+ciCDMjYypVzgmDROskoAcoVRdKyOE1ZJ3jCOPvJphEPwDSNUBGiYu6LCZdTcwMsepGOvNYbk/c9LIIyczNFh/H46cgekYgVx0i8LwmhJkxCnaK7N15NkMHsK7yInjLqfzKvZQ0z7mfmeXKVgDQVEDxjsdUYq3UUCcbA8muWyuSUtTu3+wSG/v2xhl
woody.rrze.fau.de ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPC9x4/BNwKQro3+95Gwh4DZpHBT2tVHPjKouwIBOk6g
woody.rrze.fau.de ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL7lrwFFhlZZ7mGBJ3f5gSxDEKcxvebrXLXd/bz0fH6A9Qk2GrJN2tL+sleVPRJHTboOFbdeaJy0igSwivqI2vc= 

meggie.rrze.fau.de ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwi2jQPuIe88/SJRmaKmA1VOOse4UxyjWlqp6VHM+8gggkpajGz3l6xZD1BihqOpY10oIA6rRHBQipZmFGgDkgTT40jdMvP8sLzqtJqKoQILXJqQbGWrGgjEDwXdZHIWaiV5Q8XDAgqj9+4W9ZHfeGtgS2OqhzAlTdgHzx94h8m6J8JUc+QtPGlWGBr/Z2Ee+KFEG1siT09k7E72sOnL9VDqMHFlWtHUsGfcR+8f6hnKnSHBB2TpxGac2Yv0KpqtHFdGMLY22RzDgCoEeY42fLvOqF9xIU8NgWoqII4W1AcvvpPDe8EthnKkaMsQjqj6N1uJ1qpsOZry7TiwQQF2/D
meggie.rrze.fau.de ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNmjmhh6fMxEkmNybzP3Maau/KRbOTZECKF8FxZVH3a3rMirSyjRG8LLNswctajPJxeQCAb5OIh1A63PbsIA2g8=
meggie.rrze.fau.de ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILOVhUpUyaugYDdwpHCuKfcgS0PQjZN+7KlbJ5ByZvhi 

Please keep in mind that these keys are changed from time to time. So if you get a warning while connecting, first check the above keys if they have changed!

Advanced Usage

 

Proxy Jump

If you want to connect from one host to another without the risks involved by using SSH agent forwarding and without having to type your password each time, the ProxyJump functionality of SSH can be a good alternative. When using ProxyJump, the connection is forwarded through one or more jump hosts to the target destination host via SSH. This is the most secure method because encryption is end-to-end. You can use ProxyJump for example to connect to the emmy cluster frontend by using cshpc as the jump host. This can be achieved via the following additions to ~/.ssh/config:

Host emmy
  HostName emmy.rrze.fau.de
  ProxyJump cshpc.rrze.fau.de

SSH config

There are some options to use in ~/.ssh/config that can simplify your workflow. To see all available options, type man ssh_config in your terminal.

  • Instead of defining the same identity file explicitly for every host, you can also define to always use the same key for a specific user:
    Match User JohnDoe
      IdentityFile ~/.ssh/private_ssh_key_name
  • Specify that only the SSH keys listed in ~/.ssh/config should be tried for authentication. This can avoid a „Too many authentication failures“ error, if the SSH agent offers many different keys.
    IdentitiesOnly yes
  • It is possible to use wildcards (*,?,..) in hostnames to reduce the number of explicit entries. For example, it is possible to deny SSH agent and X11 forwarding for all hosts via:
    Hosts * 
      ForwardAgent no 
      ForwardX11 no