Running Jobs

IDUN uses the Slurm Workload Manager to manage the provided resources and to schedule jobs on these resources.

Sample SLURM Job Scripts

MPI USE CASE

job.slurm (example with compiled c-code or fortran code).

#!/bin/sh
#SBATCH --partition=CPUQ
#SBATCH --account=<account>
#SBATCH --time=00:15:00
#SBATCH --nodes=2              # 2 compute nodes
#SBATCH --ntasks-per-node=1    # 1 mpi process each node
#SBATCH --mem=12000
#SBATCH --job-name="hello_test"
#SBATCH --output=test-srun.out
#SBATCH --mail-user=<email>
#SBATCH --mail-type=ALL

WORKDIR=${SLURM_SUBMIT_DIR}
cd ${WORKDIR}
echo "we are running from this directory: $SLURM_SUBMIT_DIR"
echo " the name of the job is: $SLURM_JOB_NAME"
echo "Th job ID is $SLURM_JOB_ID"
echo "The job was run on these nodes: $SLURM_JOB_NODELIST"
echo "Number of nodes: $SLURM_JOB_NUM_NODES"
echo "We are using $SLURM_CPUS_ON_NODE cores"
echo "We are using $SLURM_CPUS_ON_NODE cores per node"
echo "Total of $SLURM_NTASKS cores"

module purge
module load intel/2020b
module list
mpirun myprogram

Use same module in the script as for the compilation of the code.

Simple use case

Run on one node.

(See below for example with Matlab, Python and Comsol)

#!/bin/sh
#SBATCH --partition=CPUQ
#SBATCH --account=<account>
#SBATCH --time=00:15:00
#SBATCH --nodes=1
#SBATCH -c 28
#SBATCH --mem=12000
#SBATCH --job-name="hello_test"
#SBATCH --output=test-srun.out
#SBATCH --mail-user=<email>
#SBATCH --mail-type=ALL
 
WORKDIR=${SLURM_SUBMIT_DIR}
cd ${WORKDIR}
echo "we are running from this directory: $SLURM_SUBMIT_DIR"
echo " the name of the job is: $SLURM_JOB_NAME"
echo "Th job ID is $SLURM_JOB_ID"
echo "The job was run on these nodes: $SLURM_JOB_NODELIST"
echo "Number of nodes: $SLURM_JOB_NUM_NODES"
echo "We are using $SLURM_CPUS_ON_NODE cores"
echo "We are using $SLURM_CPUS_ON_NODE cores per node"
echo "Total of $SLURM_NTASKS cores"

module purge
module load intel/2020b
module list
./myprogram

Note, that you need to replace <email> and <account> with your email address and an allocation account, respectively. Be aware that the line with “#SBATCH –partition=CPUQ” may need to be edited to your correct partition:

CPUQ is the default partition for pure CPU jobs. If you need GPU’s, please use the GPUQ-queues

Although GPU partition also have lots of CPU resources, please don’t use those partitions for long running CPU only jobs. Prefereably stay within 30 minutes when submitting CPU jobs to GPU nodes

#SBATCH --partition=CPUQ
#SBATCH --partition=GPUQ     #GPU
#SBATCH --partition=TRAINING

Available partitions may be found with:

idun-login2$ scontrol show partition|grep ^Par
PartitionName=CPUQ
PartitionName=GPUQ
PartitionName=short

Save the script, e.g. as job.slurm, and submit the job using command sbatch:

$ chmod u+x job.slurm
$ sbatch job.slurm

Run several tasks and several nodes

If you your job is MPI and you want to run on several tasks and nodes, apply corresponding changes.

If you need to run your job on two nodes and ten tasks (cores) on each node:

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=10

GPU job with slurm

To get access to gpu resources, you need to add a request for GPU resources per node

You may choose between

P100 (54 available GPUS)

V100 (36 available GPUS)

if you need one gpu:

if you need two gpu:
#SBATCH --gres=gpu:2
If you are dependent on specific type of GPU, for example Volta GPU:
#SBATCH --gres=gpu:V100:1
Or Pascal GPU:
#SBATCH --gres=gpu:P100:1
gpu resources
#!/bin/sh
#SBATCH --partition=GPUQ
#SBATCH --account=<account>
#SBATCH --time=00:30:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
#SBATCH --gres=gpu:1  
#SBATCH --job-name="LBM_CUDA"
#SBATCH --output=lbm_cuda.out

cd ${SLURM_SUBMIT_DIR}

module purge
module load fosscuda/2018b
mpirun hostname

mpirun ./my cudacode

Array jobs:

To run many sub jobs from same job, use –array as

#SBATCH –array=1-10

Starts 10 sub jobs and you get a array ID from each sub job with environment variable $SLURM_ARRAY_TASK_ID

Example:

#!/bin/sh
#SBATCH --partition=CPUQ
#SBATCH --account=<account>
#SBATCH --time=00:15:00
#SBATCH --nodes=1
#SBATCH -c 28
#SBATCH --mem=12000
#SBATCH --array=1-10
#SBATCH --job-name="hello_test"
#SBATCH --output=test-srun.out
#SBATCH --mail-user=<email>
#SBATCH --mail-type=ALL
 
WORKDIR=${SLURM_SUBMIT_DIR}
cd ${WORKDIR}

module purge
module load intel/2020b
module list
./myprogram $SLURM_ARRAY_TASK_ID

This example will start 10 sub jobs and the running program will get an input argument with individual Task ID, which is from 1 to 10.

Other array jobs settings:

Specification   Resulting SLURM_ARRAY_TASK_IDs
1,4,42          # 1, 4, 42
1-5             # 1, 2, 3, 4, 5
0-10:2          # 0, 2, 4, 6, 8, 10 (step 2)
32,56,100-200   # 32, 56, 100, 101, 102, ..., 200
1-200%10        # 1, 2, ..., 200, but maximum 10 running at the same time

Interactive Jobs:

A user can also request node(s) for an interactive job:

Interactive Job
[<username>@idun-login1 ~]$ srun --nodes=1 --partition=CPUQ --time=00:30:00 --pty bash
srun: job 416172 queued and waiting for resources
srun: job 416172 has been allocated resources

[compute-1-0-27 ~]$

The above example requests a single node from the CPUQpartition for 30 minutes. After the allocation, the user is prompted with a bash shell on the requested node.

An alternative way to accomplish the same interactive job is:

[<username>@idun-login1 ~]$ salloc --nodes=1 --partition=CPUQ --time=00:30:00
salloc: Pending job allocation 416189
salloc: job 416189 queued and waiting for resources
salloc: job 416189 has been allocated resources
salloc: Granted job allocation 416189
salloc: Waiting for resource configuration
salloc: Nodes compute-1-0-27 are ready for job

[<username>@idun-login1 ~]$ ssh compute-1-0-27

[reissman@compute-1-0-27 ~]$

The first command reserves again a single node from the WORKQ partition for 30 minutes, but in contrast to srun, does not automatically log into the allocated node. The login is then achieved by using the ssh command.

Running Graphical User Interface (GUI) Applications:

In order to use GUI applications, it is necessary to use the -X flag when logging into the cluster:

XForwarding
$ ssh -X <username>@idun-login1.hpc.ntnu.no
<username>@idun-login1.hpc.ntnu.no's password:

[<username>@idun-login1 ~]$

This will enable the forwarding of a GUI applications’ windows to your desktop machine. After login, an interactive job can be started for the execution of the GUI application:

[<username>@idun-login1 ~] salloc --nodes=1 --partition=CPUQ --time=00:30:00
salloc: Pending job allocation 416194
salloc: job 416194 queued and waiting for resources
salloc: job 416194 has been allocated resources
salloc: Granted job allocation 416194
salloc: Waiting for resource configuration
salloc: Nodes compute-1-0-27 are ready for job

[<username>@idun-login1 ~]@ ssh -X compute-1-0-27

[<username>@compute-1-0-27 ~] xclock

The first command reserves a single node from the WORKQ partition for 30 minutes. After the allocation has been granted, it is possible to log into this node using the -X flag and start the GUI application (xclock in the above example).

Python job script example (Slurm):

Exampel Python/3.8.6-GCCcore-10.2.0. Use “module spider python” to find versions.

#!/bin/bash
#SBATCH --job-name="my-job"   # Sensible name for the job
#SBATCH --account=<account>   # Account for consumed resources
#SBATCH --nodes=1             # Allocate 1 nodes for the job
#SBATCH -c28                  # Number of cores (can vary)
#SBATCH --time=00-00:10:00    # Upper time limit for the job (DD-HH:MM:SS)
#SBATCH --partition=CPUQ

module load Python/3.8.6-GCCcore-10.2.0

python mypython.py 

For parallel python see: Parallel Python – High Performance Computing Group (ntnu.no)

Matlab job script example (Slurm):

#!/bin/bash
#SBATCH --job-name="my-job"   # Sensible name for the job
#SBATCH --account=<account>   # Account for consumed resources
#SBATCH --nodes=1             # Allocate 1 nodes for the job
#SBATCH -c28                  # Number of cores (can vary)
#SBATCH --time=00-00:10:00    # Upper time limit (DD-HH:MM:SS)
#SBATCH --partition=CPUQ

module load MATLAB/2020b

matlab -nodisplay -nodesktop -nosplash -nojvm -r "test"

(NOTE! If using Parallel Computing Toolbox, remove -nojvm as: matlab -nodisplay -nodesktop -nosplash -r “test”)

For parallel Matlab (using MPI) see: Distributed Matlab (using MPI) – High Performance Computing Group (ntnu.no)

COMSOL job script example (Slurm):

#!/bin/bash
########################################################
#
#  Running COMSOL cluster job
#
#  The default setup is to run COMSOL in hybrid mode
#  with 2 MPI processes per node ('mpiprocs'), one per 
#  socket, and 8 threads ('ompthreads') per MPI process
#
########################################################
#
#SBATCH --partition=CPUQ    # partition the batch job will be put in
#SABTCH --account=<account>  # Account for consumed resources
#SBATCH --time=00:30:00      # the walltime length of the job (30 minutes)
#SBATCH --nodes=2            # number of nodes requested
#SBATCH --ntasks-per-node=2  # number of processes per node
#SBATCH --job-name="comsol_example"  # name of the job
#SBATCH --output=comsol.out  # name of output file
#SBATCH -J COMSOL_example   # Name for the job 
 
module load COMSOL/5.3a


# ${SLURM_SUBMIT_DIR} - directory from where the job were submitted 
# Create (if necessary) the working directory
w=${SLURM_SUBMIT_DIR}/comsol/$case
if [ ! -d $w ]; then mkdir -p $w; fi
 
# Copy inputfile and move to working directory
cp $case.mph $w
cd $w
 
comsol batch -mpibootstrap ssh -inputfile $case.mph -outputfile out.mph -tmpdir $w

 

Controlling Slurm jobs

# Get all jobs
squeue
 
# get all jobs for user < only pending | only running > in <partition>
squeue -u username <-t PENDING|-t RUNNING> <-p partition>
 
# Show detailed info on <jobid>
scontrol show jobid -dd <jobid>
 
# cancel specific <jobid>
scancel < jobid >
 
# cancel all <pending> jobs for <username>
scancel <-t PENDING> -u <username>