IDUN uses the Slurm Workload Manager to manage the provided resources and to schedule jobs on these resources.
NOTE 1: Max Walltime for Idun is 7 days or 167 hours.
NOTE 2: Use partition "short" to test your scripts and jobs. "short" has 4 servers with P100 GPUs.
If you need more, start your job with 7 days and send an request to help desk, with the job id, about more hours for this specific job.
Sample SLURM Job Scripts
Simple use case
Run on one node.
(See below for example with Matlab, Python and Comsol)
#!/bin/sh
#SBATCH --partition=CPUQ
#SBATCH --account=<account>
#SBATCH --time=00:15:00
#SBATCH --nodes=1 # 1 compute nodes
#SBATCH --cpus-per-task=2 # 2 CPU cores
#SBATCH --mem=12000
#SBATCH --job-name="hello_test"
#SBATCH --output=test-srun.out
#SBATCH --mail-user=<email>
#SBATCH --mail-type=ALL
WORKDIR=${SLURM_SUBMIT_DIR}
cd ${WORKDIR}
echo "we are running from this directory: $SLURM_SUBMIT_DIR"
echo " the name of the job is: $SLURM_JOB_NAME"
echo "Th job ID is $SLURM_JOB_ID"
echo "The job was run on these nodes: $SLURM_JOB_NODELIST"
echo "Number of nodes: $SLURM_JOB_NUM_NODES"
echo "We are using $SLURM_CPUS_ON_NODE cores"
echo "We are using $SLURM_CPUS_ON_NODE cores per node"
echo "Total of $SLURM_NTASKS cores"
module purge
module load intel/2020b
module list
./myprogram
Note, that you need to replace <email> and <account> with your email address and an allocation account, respectively. Be aware that the line with "#SBATCH --partition=CPUQ" may need to be edited to your correct partition:
CPUQ is the default partition for pure CPU jobs. If you need GPU's, please use the GPUQ-queues
Although GPU partition also have lots of CPU resources, please don't use those partitions for long running CPU only jobs. Prefereably stay within 30 minutes when submitting CPU jobs to GPU nodes
#SBATCH --partition=CPUQ #SBATCH --partition=GPUQ #GPU
Available partitions may be found with:
idun-login2$ scontrol show partition|grep ^Par PartitionName=CPUQ PartitionName=GPUQ PartitionName=short
Save the script, e.g. as job.slurm, and submit the job using command sbatch:
$ chmod u+x job.slurm $ sbatch job.slurm
MPI USE CASE
job.slurm (example with compiled c-code or fortran code).
#!/bin/sh
#SBATCH --partition=CPUQ
#SBATCH --account=<account>
#SBATCH --time=00:15:00
#SBATCH --nodes=2 # 2 compute nodes
#SBATCH --ntasks-per-node=1 # 1 mpi process each node
#SBATCH --mem=12000 # 12GB - in megabytes
#SBATCH --job-name="hello_test"
#SBATCH --output=test-srun.out
#SBATCH --mail-user=<email>
#SBATCH --mail-type=ALL
WORKDIR=${SLURM_SUBMIT_DIR}
cd ${WORKDIR}
echo "we are running from this directory: $SLURM_SUBMIT_DIR"
echo " the name of the job is: $SLURM_JOB_NAME"
echo "Th job ID is $SLURM_JOB_ID"
echo "The job was run on these nodes: $SLURM_JOB_NODELIST"
echo "Number of nodes: $SLURM_JOB_NUM_NODES"
echo "We are using $SLURM_CPUS_ON_NODE cores"
echo "We are using $SLURM_CPUS_ON_NODE cores per node"
echo "Total of $SLURM_NTASKS cores"
module purge
module load intel/2020b
module list
srun myprogram
Use same module in the script as for the compilation of the code.
Run several tasks and several nodes
If you your job is MPI and you want to run on several tasks and nodes, apply corresponding changes.
If you need to run your job on two nodes and ten tasks (cores) on each node:
#SBATCH --nodes=2 #SBATCH --ntasks-per-node=10
GPU job with slurm
To get access to gpu resources, you need to add a request for GPU resources per node. You may choose between:
- P100 (40 available GPUS) with 16GB
- V100 (38 available GPUS) with 16GB and 32GB
- A100 (64 available GPUS) with 40GB and 80GB
if you need one gpu:
#SBATCH --gres=gpu:1
if you need two gpu:
#SBATCH --gres=gpu:2
If you are dependent on specific type of GPU (P100,V100,A100), for example Volta GPU:
#SBATCH --gres=gpu:1 #SBATCH --constraint=V100
If you are dependent on specific GPU memory size. Use this options:
16G P100 16G V100 32G V10032 40G A100m40 80G A100m80
In script:
#SBATCH --gres=gpu:A100m40:1
It is also possible combine constraints with logical AND "&", logical OR "|":
#SBATCH --constraint="V100|A100"#SBATCH --gres=gpu:2
Example:
#!/bin/sh #SBATCH --partition=GPUQ #SBATCH --account=<account> #SBATCH --time=00:30:00 #SBATCH --nodes=2 #SBATCH --ntasks-per-node=2 #SBATCH --gres=gpu:2 #SBATCH --job-name="LBM_CUDA" #SBATCH --output=lbm_cuda.out module purge module load fosscuda/2018b mpirun hostname srun ./my cudacode
Array jobs:
To run many sub jobs from same job, use --array as
#SBATCH --array=1-10
Starts 10 sub jobs and you get a array ID from each sub job with environment variable $SLURM_ARRAY_TASK_ID
Example:
#!/bin/sh
#SBATCH --partition=CPUQ
#SBATCH --account=<account>
#SBATCH --time=00:15:00
#SBATCH --nodes=1
#SBATCH -c 28
#SBATCH --mem=12000
#SBATCH --array=1-10
#SBATCH --job-name="hello_test"
#SBATCH --output=test-srun.out
#SBATCH --mail-user=<email>
#SBATCH --mail-type=ALL
WORKDIR=${SLURM_SUBMIT_DIR}
cd ${WORKDIR}
module purge
module load intel/2020b
module list
./myprogram $SLURM_ARRAY_TASK_ID
This example will start 10 sub jobs and the running program will get an input argument with individual Task ID, which is from 1 to 10.
Other array jobs settings:
Specification Resulting SLURM_ARRAY_TASK_IDs 1,4,42 # 1, 4, 42 1-5 # 1, 2, 3, 4, 5 0-10:2 # 0, 2, 4, 6, 8, 10 (step 2) 32,56,100-200 # 32, 56, 100, 101, 102, ..., 200 1-200%10 # 1, 2, ..., 200, but maximum 10 running at the same time
Interactive Jobs:
A user can also request node(s) for an interactive job:
Interactive Job[<username>@idun-login1 ~]$ srun --account=<account> --nodes=1 --partition=CPUQ --time=00:30:00 --pty bash srun: job 416172 queued and waiting for resources srun: job 416172 has been allocated resources [compute-1-0-27 ~]$
The above example requests a single node from the CPUQpartition for 30 minutes. After the allocation, the user is prompted with a bash shell on the requested node.
An alternative way to accomplish the same interactive job is:
[<username>@idun-login1 ~]$ salloc --account=<account> --nodes=1 --partition=CPUQ --time=00:30:00 salloc: Pending job allocation 416189 salloc: job 416189 queued and waiting for resources salloc: job 416189 has been allocated resources salloc: Granted job allocation 416189 salloc: Waiting for resource configuration salloc: Nodes compute-1-0-27 are ready for job [<username>@idun-login1 ~]$ ssh compute-1-0-27 [reissman@compute-1-0-27 ~]$
The first command reserves again a single node from the WORKQ partition for 30 minutes, but in contrast to srun, does not automatically log into the allocated node. The login is then achieved by using the ssh command.
Running Graphical User Interface (GUI) Applications:
In order to use GUI applications, it is necessary to use the -X flag when logging into the cluster:
XForwarding$ ssh -X <username>@idun-login1.hpc.ntnu.no <username>@idun-login1.hpc.ntnu.no's password: [<username>@idun-login1 ~]$
This will enable the forwarding of a GUI applications' windows to your desktop machine. After login, an interactive job can be started for the execution of the GUI application:
[<username>@idun-login1 ~] salloc --account=<account> --nodes=1 --partition=CPUQ --time=00:30:00 salloc: Pending job allocation 416194 salloc: job 416194 queued and waiting for resources salloc: job 416194 has been allocated resources salloc: Granted job allocation 416194 salloc: Waiting for resource configuration salloc: Nodes compute-1-0-27 are ready for job [<username>@idun-login1 ~]@ ssh -X compute-1-0-27 [<username>@compute-1-0-27 ~] xclock
The first command reserves a single node from the WORKQ partition for 30 minutes. After the allocation has been granted, it is possible to log into this node using the -X flag and start the GUI application (xclock in the above example).
Python job script example (Slurm):
Exampel Python/3.8.6-GCCcore-10.2.0. Use "module spider python" to find versions.
#!/bin/bash #SBATCH --job-name="my-job" # Sensible name for the job #SBATCH --account=<account> # Account for consumed resources #SBATCH --nodes=1 # Allocate 1 nodes for the job #SBATCH -c28 # Number of cores (can vary) #SBATCH --time=00-00:10:00 # Upper time limit for the job (DD-HH:MM:SS) #SBATCH --partition=CPUQ module load Python/3.8.6-GCCcore-10.2.0 python mypython.py
For parallel python see: Parallel Python – High Performance Computing Group (ntnu.no)
Matlab job script example (Slurm):
#!/bin/bash #SBATCH --job-name="my-job" # Sensible name for the job #SBATCH --account=<account> # Account for consumed resources #SBATCH --nodes=1 # Allocate 1 nodes for the job #SBATCH -c28 # Number of cores (can vary) #SBATCH --time=00-00:10:00 # Upper time limit (DD-HH:MM:SS) #SBATCH --partition=CPUQ module load MATLAB/2021b matlab -nodisplay -nodesktop -nosplash -nojvm -r "test"
(NOTE! If using Parallel Computing Toolbox, remove -nojvm as: matlab -nodisplay -nodesktop -nosplash -r "test")
To check Matlab versions, type: module spider matlab
For parallel Matlab (using MPI) see: Distributed Matlab(using MPI) – High Performance Computing Group (ntnu.no)
COMSOL job script example (Slurm):
#!/bin/bash ######################################################## # # Running COMSOL cluster job # # The default setup is to run COMSOL in hybrid mode # with 2 MPI processes per node ('mpiprocs'), one per # socket, and 8 threads ('ompthreads') per MPI process # ######################################################## # #SBATCH --partition=CPUQ # partition the batch job will be put in #SABTCH --account=<account> # Account for consumed resources #SBATCH --time=00:30:00 # the walltime length of the job (30 minutes) #SBATCH --nodes=2 # number of nodes requested #SBATCH --ntasks-per-node=2 # number of processes per node #SBATCH --job-name="comsol_example" # name of the job #SBATCH --output=comsol.out # name of output file #SBATCH -J COMSOL_example # Name for the job module load COMSOL/5.3a # ${SLURM_SUBMIT_DIR} - directory from where the job were submitted # Create (if necessary) the working directory w=${SLURM_SUBMIT_DIR}/comsol/$case if [ ! -d $w ]; then mkdir -p $w; fi # Copy inputfile and move to working directory cp $case.mph $w cd $w comsol batch -mpibootstrap ssh -inputfile $case.mph -outputfile out.mph -tmpdir $w
Controlling Slurm jobs
# Get all jobs squeue # get all jobs for user < only pending | only running > in <partition> squeue -u username <-t PENDING|-t RUNNING> <-p partition> # Show detailed info on <jobid> scontrol show jobid -dd <jobid> # cancel specific <jobid> scancel < jobid > # cancel all <pending> jobs for <username> scancel <-t PENDING> -u <username>