Tutorial - Using GPUs on Tegner

Connect to Tegner

Connect to Tegner by setting up Kerberos and SSH into the login node.

kinit -f (your username)@NADA.KTH.SE
ssh -Y (your username)@tegner.pdc.kth.se

Follow Tutorial - Connecting to PDC Supercomputer for more information.

---

Prepare Environment

Prepare the environment by loading the necessary modules:

module load gcc/6.2.0
module load openmpi/3.0-gcc-6.2
module load cuda/10.0

To use the PGI compiler with OpenACC, additionally do:

module load pgi

Refer to here for more information about modules on Tegner.

---

Request a node

Request compute nodes using the usual salloc command, but append a special flag --gres to specify the GPUs:

salloc --nodes=1 -C Haswell --gres=gpu:K420:1 -t 00:05:00 -A edu19.DD2360 --reservation=reservation_here

We specify -C Haswell --gres=gpu:K420:1 to ask for thin nodes and each with one K420 GPU.

To ask for a fat node with two K420 GPUs, set --gres=K420:2 to specify two GPUs, and omit -C Haswell:

salloc --nodes=1 --gres=gpu:K420:2 -t 00:05:00 -A edu19.DD2360 --reservation=reservation_here

Alternatively, we can ask for K80 GPUs with:

salloc --nodes=1 -C Haswell --gres=gpu:K80:2 -t 00:05:00 -A edu19.DD2360 --reservation=reservation_here

We specify --gres=gpu:K80:2 as there are two GPU engines per K80 card, which can be programmed as two separate GPUs. We specify 2 to get both engines.

---

Compile and run programs

Ensure that you are in cfs:

cd /cfs/klemming/nobackup/(initial of username)/(username)/

Compile the program as usual:

nvcc -arch=sm_30 program.cu -o program.out

Run your program through the batch system:

srun -n 1 ./program.out

Use -arch=sm_30 for K420 and -arch=sm_37 for K80. Refer to lecture on how to compile a fat binary that supports multiple architecture.