Introduction
The TinyX-clusters are a group of small special purpose clusters:
- TinyFat for applications needing lots of main memory
- TinyGPU for applications utilizing GPU-accelerators
- TinyEth for throughput workloads – this cluster is made from „recycled“ hardware
While their hardware is very diverse, these clusters share a few properties:
- They are very small, hence the TinySomething naming scheme.
- They do not have their own frontends. Access is through the Woody frontends.
- They have local harddiscs (other than the big clusters which run diskless).
In general, the Documentation for Woody applies, this page will only list the differences.
Click the header to show the description for each cluster:
| Hostnames | # nodes |
CPUs and number of cores per machine | Main memory (in GB) |
Additional comments and required qsub parameters |
|---|---|---|---|---|
| tf001-tf016 | 15 | 2x AMD Opteron 6134 („Magny Cours“) @2,3 GHz = 16 cores |
128 | These nodes have QDR Infiniband between them.-l nodes=n:ppn=16 with n between 1 and 8 |
| memoryhog (formerly tf020) |
1 | 4x Intel Xeon X7560 („Nehalem EX“) @2,27 GHz = 32 cores/64 threads |
512 | interactively accesible without batch job |
| tf030 | 1 | 4x AMD Opteron 6176 SE („Magny Cours“) @2,3 GHz = 48 cores |
192 | accesible on request only |
| tf040-tf042 (not generally available) |
3 | 2x Intel Xeon E5-2680 v4 („Broadwell“) @2,4 GHz = 28 cores/56 threads |
512 | 10 GBit Ethernet, 1 TB Intel 750 SSD-q broadwell512 -l nodes=1:ppn=56 |
| tf050-tf057 (not generally available) |
7 | 2x Intel Xeon E5-2643 v4 („Broadwell“) @3,4 GHz = 12 cores/24 threads |
256 | 10 GBit Ethernet, 1 TB Intel 750 SSD-q broadwell256 -l nodes=1:ppn=24 |
To submit batch jobs to TinyFat, you need to use qsub.tinyfat instead of the normal qsub-command on the Woody frontends. Be sure to use the correct parameters for qsub as specified in the table above.
To check the status of your jobs, use qstat.tinyfat instead of the normal qstat.
In the nodes that have an SSD, it is mounted in the directory /scratchssd. Please only use these if you can really profit from their use, as like all consumer SSDs they only support a limited number of writes, so in other words, by writing to them, you „use them up“.
All Broadwell-based nodes have been purchased by specific groups or special projects. These uses have priority access and nodes may be reserved exclusively for them.
There is also the machine memoryhog that is somewhat part of the cluster: Every HPC user can log in directly to memoryhog.rrze.uni-erlangen.de to run their memory intensive workloads. This of course means you need to be considerate of other users.
Processes hogging up too many resources or running for too long will be killed without notice.
| Hostnames | # nodes |
CPUs and number of cores per machine, main memory | GPU Type | Additional comments |
|---|---|---|---|---|
| tg001-tg008 | 7 | 2x Intel Xeon 5550 („Nehalem“) @2.66 Ghz = 8 cores/SMT off; 24 GB RAM |
2x NVIDIA GTX 980 | |
| tg031-tg037 | 7 | 2x Intel Xeon E5-2620v4 („Broadwell“) @2.1 GHz = 16 cores/SMT off; 64 GB RAM |
4x NVIDIA GTX 1080 | local SATA-SSD (880 GB) available under /scratchssd |
| tg040-tg049 (not generally available) |
10 | 2x Intel Xeon E5-2620v4 („Broadwell“) @2.1 GHz = 16 cores/SMT off; 64 GB RAM (one node has 128 GB RAM) |
4x NVIDIA GTX 1080 Ti | local SATA-SSD (1.8 TB) available under /scratchssd |
| tg071-tg073 | 3 | 2x Intel Xeon Gold 6134 („Skylake“) @3.2 GHz = 16 cores/SMT off; 96 GB RAM |
4x NVIDIA Tesla V100 (two nodes with 4x 32GB, one node with 4x 16 GB) | local NVMe-SSD (2.9 TB) available under /scratchssd |
To submit batch jobs to TinyGPU, you need to use qsub.tinygpu instead of the normal qsub-command on the Woody frontends.
If you do not request a specific GPU type, your job will run on any available node. If you want to request a specific GPU type, use the properties :gtx980, :gtx1080, :gtx1080ti or :any1080 = gtx1080 or gtx1080ti, and :v100 or specifically :v100_16g and :v100_32g. In addition there are :anygtx and :anytesla (their meaning should be obvious) as well as :cuda8, :cuda9, and :cuda10 which tell the supported CUDA versions. Thus, e.g. qsub.tinygpu -l nodes=1:ppn=8:gtx980 [...].
You may request parts of a node, e.g. if you only need one GPU. For each 4 cores you request, you are also assigned one GPU. For obvious reasons, you are only allowed to request multiples of 4 cores. As an example, if you request qsub.tinygpu -l nodes=1:ppn=8:gtx1080 [...] you will get 8 cores, 2 of the GTX 1080 GPUs and half of the main memory in one of the tg03X nodes. There is no dedicated share of the local HDD/SDD assigned.
To check the status of your jobs, use qstat.tinygpu instead of the normal qstat.
The Woody frontends only have a limited software installation with regard to GPGPU computing. It is recommended to compile code on one of the TinyGPU nodes, i.e. by requesting an interactive job on TinyGPU.
cuDNN is not provided by RRZE due to licensing issues. If needed, download your personal copy yourself from the NVidia portal.
The GTX1080/GTX1080Ti GPUs can only be used with CUDA 8.0 (or higher). The V100 may require at least CUDA 9.0.
Host software compiled on the new tg07x or tg03x/tg04x nodes might not run on the older tg00x nodes if AVX/AVX2 instructions are used.
21 out of the 27 nodes have been purchased by specific groups or special projects. These uses have priority access and nodes may be reserved exclusively for them.
All nodes are equipped with 2x Xeon 5650 („Westmere“) CPUs (12 cores, SMT disabled) running at 2.66 GHz and 48 GB of RAM. They have a small and slow local hard disc.
To submit batch jobs to TinyEth, you need to use qsub.tinyeth instead of the normal qsub-command on the Woody frontends. The request string is-l nodes=1:ppn=X with X being any number between 1 and 12. If you do not request a full node (i.e. X<12 you only get a corresponding fraction of the main memory. To check the status of your jobs, use qstat.tinyeth instead of the normal qstat.