Running Jobs

IDUN uses the Slurm Workload Manager to manage the provided resources and to schedule jobs on these resources.

Sample SLURM Job Scripts

Simple use case

job.slurm
#!/bin/sh
#SBATCH --partition=CPUQ
#SBATCH --account=<account>
#SBATCH --time=00:15:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --mem=12000
#SBATCH --job-name="hello_test"
#SBATCH --output=test-srun.out
#SBATCH --mail-user=<email>
#SBATCH --mail-type=ALL
 
WORKDIR=${SLURM_SUBMIT_DIR}
cd ${WORKDIR}
echo "we are running from this directory: $SLURM_SUBMIT_DIR"
echo " the name of the job is: $SLURM_JOB_NAME"
echo "Th job ID is $SLURM_JOB_ID"
echo "The job was run on these nodes: $SLURM_JOB_NODELIST"
echo "Number of nodes: $SLURM_JOB_NUM_NODES"
echo "We are using $SLURM_CPUS_ON_NODE cores"
echo "We are using $SLURM_CPUS_ON_NODE cores per node"
echo "Total of $SLURM_NTASKS cores"

module purge
module load foss/2018b
mpirun hostname

uname -a

Note, that you need to replace <email> and <account> with your email address and an allocation account, respectively. Be aware that the line with “#SBATCH –partition=CPUQ” may need to be edited to your correct partition:

CPUQ is the default partition for pure CPU jobs. If you need GPU’s, please use the EPIC-queues

Although GPU partition also have lots of CPU resources, please don’t use those partitions for long running CPU only jobs. Prefereably stay within 30 minutes when submitting CPU jobs to GPU nodes

#SBATCH --partition=CPUQ
#SBATCH --partition=GPUQ     #GPU
#SBATCH --partition=TRAINING

Available partitions may be found with:

idun-login2$ scontrol show partition|grep ^Par
PartitionName=TRAINING
PartitionName=EPT
PartitionName=WORKQ
PartitionName=TEST
PartitionName=EPIC
PartitionName=EPIC2
PartitionName=EPICALL

Save the script, e.g. as job.slurm, and submit the job using command sbatch:

$ chmod u+x job.slurm
$ sbatch job.slurm

GPU job with slurm

To get access to gpu resources, you need to add a request for GPU resources per node

You may choose between

P100 (54 available GPUS)

V100 (36 available GPUS)

if you need one gpu:

#SBATCH --gres=gpu:1
if you need two gpu:
#SBATCH --gres=gpu:2
If you are dependent on specific type of GPU, for example Volta GPU:
#SBATCH --gres=gpu:V100:1
Or Pascal GPU:
#SBATCH --gres=gpu:P100:1
gpu resources
#!/bin/sh
#SBATCH --partition=GPUQ
#SBATCH --account=<account>
#SBATCH --time=00:30:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
#SBATCH --gres=gpu:1  
#SBATCH --job-name="LBM_CUDA"
#SBATCH --output=lbm_cuda.out

cd ${SLURM_SUBMIT_DIR}

module purge
module load fosscuda/2018b
mpirun hostname

mpirun ./my cudacode

Interactive Jobs:

A user can also request node(s) for an interactive job:

Interactive Job
[<username>@idun-login1 ~]$ srun --nodes=1 --partition=CPUQ --time=00:30:00 --pty bash
srun: job 416172 queued and waiting for resources
srun: job 416172 has been allocated resources

[compute-1-0-27 ~]$

The above example requests a single node from the CPUQpartition for 30 minutes. After the allocation, the user is prompted with a bash shell on the requested node.

An alternative way to accomplish the same interactive job is:

[<username>@idun-login1 ~]$ salloc --nodes=1 --partition=CPUQ --time=00:30:00
salloc: Pending job allocation 416189
salloc: job 416189 queued and waiting for resources
salloc: job 416189 has been allocated resources
salloc: Granted job allocation 416189
salloc: Waiting for resource configuration
salloc: Nodes compute-1-0-27 are ready for job

[<username>@idun-login1 ~]$ ssh compute-1-0-27

[reissman@compute-1-0-27 ~]$

The first command reserves again a single node from the WORKQ partition for 30 minutes, but in contrast to srun, does not automatically log into the allocated node. The login is then achieved by using the ssh command.

Running Graphical User Interface (GUI) Applications:

In order to use GUI applications, it is necessary to use the -X flag when logging into the cluster:

XForwarding
$ ssh -X <username>@idun-login1.hpc.ntnu.no
<username>@idun-login1.hpc.ntnu.no's password:

[<username>@idun-login1 ~]$

This will enable the forwarding of a GUI applications’ windows to your desktop machine. After login, an interactive job can be started for the execution of the GUI application:

[<username>@idun-login1 ~] salloc --nodes=1 --partition=CPUQ --time=00:30:00
salloc: Pending job allocation 416194
salloc: job 416194 queued and waiting for resources
salloc: job 416194 has been allocated resources
salloc: Granted job allocation 416194
salloc: Waiting for resource configuration
salloc: Nodes compute-1-0-27 are ready for job

[<username>@idun-login1 ~]@ ssh -X compute-1-0-27

[<username>@compute-1-0-27 ~] xclock

The first command reserves a single node from the WORKQ partition for 30 minutes. After the allocation has been granted, it is possible to log into this node using the -X flag and start the GUI application (xclock in the above example).

Matlab job script example (Slurm):

#!/bin/bash
#SBATCH --job-name="my-job"   # Sensible name for the job
#SBATCH --account=<account>   # Account for consumed resources
#SBATCH --nodes=1             # Allocate 1 nodes for the job
#SBATCH --cpus-per-task=20
#SBATCH --time=00-00:10:00    # Upper time limit for the job (DD-HH:MM:SS)
#SBATCH --partition=CPUQ

module load MATLAB/2017a

matlab -nodisplay -nodesktop -nosplash -nojvm -r "test"

(NOTE! If using Parallel Computing Toolbox, remove -nojvm as: matlab -nodisplay -nodesktop -nosplash -r “test”)

COMSOL job script example (Slurm):

#!/bin/bash
########################################################
#
#  Running COMSOL cluster job
#
#  The default setup is to run COMSOL in hybrid mode
#  with 2 MPI processes per node ('mpiprocs'), one per 
#  socket, and 8 threads ('ompthreads') per MPI process
#
########################################################
#
#SBATCH --partition=CPUQ    # partition the batch job will be put in
#SABTCH --account=<account>  # Account for consumed resources
#SBATCH --time=00:30:00      # the walltime length of the job (30 minutes)
#SBATCH --nodes=2            # number of nodes requested
#SBATCH --ntasks-per-node=2  # number of processes per node
#SBATCH --job-name="comsol_example"  # name of the job
#SBATCH --output=comsol.out  # name of output file
#SBATCH -J COMSOL_example   # Name for the job 
 
module load COMSOL/5.3a


# ${SLURM_SUBMIT_DIR} - directory from where the job were submitted 
# Create (if necessary) the working directory
w=${SLURM_SUBMIT_DIR}/comsol/$case
if [ ! -d $w ]; then mkdir -p $w; fi
 
# Copy inputfile and move to working directory
cp $case.mph $w
cd $w
 
comsol batch -mpibootstrap ssh -inputfile $case.mph -outputfile out.mph -tmpdir $w

 

Controlling Slurm jobs

# Get all jobs
squeue
 
# get all jobs for user < only pending | only running > in <partition>
squeue -u username <-t PENDING|-t RUNNING> <-p partition>
 
# Show detailed info on <jobid>
scontrol show jobid -dd <jobid>
 
# cancel specific <jobid>
scancel < jobid >
 
# cancel all <pending> jobs for <username>
scancel <-t PENDING> -u <username>