Introduction
The TinyX-clusters are a group of small special purpose clusters:
- TinyFat for applications needing lots of main memory
- TinyGPU for applications utilizing GPU-accelerators
- TinyEth for throughput workloads – this cluster is made from „recycled“ hardware
While their hardware is very diverse, these clusters share a few properties:
- They are very small, hence the TinySomething naming scheme.
- They do not have their own frontends. Access is through the Woody frontends.
- They have local harddiscs (other than the big clusters which run diskless).
In general, the Documentation for Woody applies, this page will only list the differences.
Click the header to show the description for each cluster:
Memoryhog and the TinyFat cluster are intended for running serial or moderately parallel (OpenMP) applications that require large amounts of memory in one machine. They are mostly used for pre-/postprocessing-work for jobs on other clusters. There are a bunch of different machines in TinyFat:
| Hostnames | # nodes |
CPUs and number of cores per machine | Main memory (in GB) |
Additional comments and required qsub parameters |
|---|---|---|---|---|
| tf001-tf016 | 15 | 2x AMD Opteron 6134 („Magny Cours“) @2,3 GHz = 16 cores |
128 | These nodes have QDR Infiniband between them.-l nodes=n:ppn=16 with n between 1 and 8 |
| memoryhog (formerly tf020) |
1 | 4x Intel Xeon X7560 („Nehalem EX“) @2,27 GHz = 32 cores/64 threads |
512 | interactively accesible without batch job |
| tf030 | 1 | 4x AMD Opteron 6176 SE („Magny Cours“) @2,3 GHz = 48 cores |
192 | accesible on request only |
| tf040-tf042 (not generally available) |
3 | 2x Intel Xeon E5-2680 v4 („Broadwell“) @2,4 GHz = 28 cores/56 threads |
512 | 10 GBit Ethernet, 1 TB Intel 750 SSD-q broadwell512 -l nodes=1:ppn=56 |
| tf050-tf056 (not generally available) |
7 | 2x Intel Xeon E5-2643 v4 („Broadwell“) @3,4 GHz = 12 cores/24 threads |
256 | 10 GBit Ethernet, 1 TB Intel 750 SSD-q broadwell256 -l nodes=1:ppn=24 |
To submit batch jobs to TinyFat, you need to use qsub.tinyfat instead of the normal qsub-command on the Woody frontends. Be sure to use the correct parameters for qsub as specified in the table above.
To check the status of your jobs, use qstat.tinyfat instead of the normal qstat.
In the nodes that have an SSD, it is mounted in the directory /scratchssd. Please only use these if you can really profit from their use, as like all consumer SSDs they only support a limited number of writes, so in other words, by writing to them, you „use them up“.
All Broadwell-based nodes have been purchased by specific groups or special projects. These uses have priority access and nodes may be reserved exclusively for them.
There is also the machine memoryhog that is somewhat part of the cluster: Every HPC user can log in directly to memoryhog.rrze.uni-erlangen.de to run their memory intensive workloads. This of course means you need to be considerate of other users.
Processes hogging up too many resources or running for too long will be killed without notice.
TinyGPU complements the GPU nodes in Emmy cluster and has nodes with three different types of consumer GPUs:
| Hostnames | # nodes |
CPUs and number of cores per machine, main memory | GPU Type | Additional comments |
|---|---|---|---|---|
| tg001-tg008 | 7 | 2x Intel Xeon 5550 („Nehalem“) @2.66 Ghz = 8 cores/16 threads; 24 GB RAM |
2x NVIDIA GTX 980 | |
| tg031-tg037 | 7 | 2x Intel Xeon E5-2620v4 („Broadwell“) @2.1 GHz = 16 cores/SMT off); 64 GB RAM |
4x NVIDIA GTX 1080 | small local SSD available under /scratchssd |
| tg040-tg048 (not generally available) |
8 | 2x Intel Xeon E5-2620v4 („Broadwell“) @2.1 GHz = 16 cores/SMT off); 64 GB RAM (one node has 128 GB RAM) |
4x NVIDIA GTX 1080 Ti | small local SSD available under /scratchssd |
To submit batch jobs to TinyGPU, you need to use qsub.tinygpu instead of the normal qsub-command on the Woody frontends.
If you do not request a specific GPU type, your job will run on any available node. If you want to request a specific GPU type, use the properties :gtx980, :gtx1080, :gtx1080ti or :any1080 = gtx1080 or gtx1080ti, e.g. qsub.tinygpu -l nodes=1:ppn=8:gtx980 [...].
You may request parts of a node, e.g. if you only need one GPU. For each 4 cores you request, you are also assigned one GPU. For obvious reasons, you are only allowed to request multiples of 4 cores. As an example, if you request qsub.tinygpu -l nodes=1:ppn=8:gtx1080 [...] you will get 8 cores and 2 of the GTX 1080 GPUs in one of the tg03X nodes.
To check the status of your jobs, use qstat.tinygpu instead of the normal qstat.
The Woody frontends only have a limited software installation with regard to GPGPU computing. It is recommended to compile code on one of the TinyGPU nodes, i.e. by requesting an interactive job on TinyGPU.
cuDNN is not provided by RRZE due to licensing issues. If needed, download your personal copy yourself from the NVidia portal.
The GTX1080/GTX1080Ti GPUs can only be used with CUDA 8.0 (or higher).
Host software compiled on the new tg03x/tg04x nodes might not run on the older tg00x nodes if AVX/AVX2 instructions are used.
17 out of the 22 nodes have been purchased by specific groups or special projects. These uses have priority access and nodes may be reserved exclusively for them.
The TinyEth-cluster is a simple cluster for throughput workloads. It consists of old LiMa-nodes where the Infiniband-port died, so they cannot be used in LiMa anymore, but they’re still good for single node jobs.
All nodes are equipped with 2x Xeon 5650 („Westmere“) CPUs (12 cores, SMT disabled) running at 2.66 GHz and 48 GB of RAM. They have a small and slow local hard disc.
To submit batch jobs to TinyEth, you need to use qsub.tinyeth instead of the normal qsub-command on the Woody frontends. You always need to request -l nodes=1:ppn=12. To check the status of your jobs, use qstat.tinyeth instead of the normal qstat.