Apptainer (former Singularity)

Table of Contents

    Apptainer/Singularity is the widely used container system for HPC. You can use Docker container to create Apptainer containers.

    Apptainer documentation is here:

    Create on login node

    NOTE: There limitation to build containers on most network filesystems:
    Most network filesystems (NFS/Lustre/GPFS etc.) do not support this uid/gid mapping in a user namespace. Because the fileserver is not aware of the mappings it will deny many operations, with ‘permission denied’ errors.

    Workaround: build containers on local drive /localscratch/ and then move container to shared filesystem.

    You can create apptainer on your computer and upload it to the IDUN. But this example shows how to create an Apptainer container on IDUN login node.

    The goal: I have a python script "" and I need to run it on Ubuntu Linux for some reason 🙂

    Create sandbox container and install missing software.

    On login node you need to use options "--no-mount hostfs" to ignore other filesystems on login nodes. And you need to use option "--fakeroot" that allows an unprivileged user to run a container with the appearance of running as root.

    cd /localscratch/
    apptainer build --sandbox myapp1 docker://ubuntu
    apptainer shell --no-mount hostfs --fakeroot --writable myapp1
    > apt update
    > apt install python3
    > apt install python3-pip
    > pip3 install pandas
    > exit

    Convert sandbox container to SIF format and run test

    apptainer build --fakeroot myapp1.sif myapp1
    rm -rf myapp1
    mv myapp1.sif /cluster/home/USERNAME/
    cd /cluster/home/USERNAME/
    apptainer run myapp1.sif python3 -V
    Python 3.10.6

    This is python script and Slurm job file

    Python script:

    import pandas as pd
    data = pd.DataFrame({"x1":["y", "x", "y", "x", "x", "y"],
                         "x2":range(16, 22),
                         "x3":range(1, 7),
                         "x4":["a", "b", "c", "d", "e", "f"],
                         "x5":range(30, 24, - 1)})

    Slurm job file: apptainer.slurm

    #SBATCH --partition=CPUQ
    #SBATCH --account=<Your_Account>
    #SBATCH --time=0-00:15:00     # 0 days and 15 minutes limit
    #SBATCH --nodes=1             # 1 compute nodes
    #SBATCH --cpus-per-task=1     # 1 CPU cores
    #SBATCH --mem=5G              # 5 gigabytes memory
    #SBATCH --output=apptainer.txt
    apptainer exec myapp1.sif python3

    Run the job and check the result

    $ sbatch apptainer.slurm 
    Submitted batch job 18636396

    Wait and check job status:

    $ scontrol show job 18636396
    . . .
       JobState=COMPLETED Reason=None Dependency=(null)

    Check output log file:

    $ cat apptainer.txt
      x1  x2  x3 x4  x5
    0  y  16   1  a  30
    1  x  17   2  b  29
    2  y  18   3  c  28
    3  x  19   4  d  27
    4  x  20   5  e  26
    5  y  21   6  f  25

    Building containers from Apptainer definition files

    Create definition file:

    $ nano lolcow.def
    Bootstrap: docker
    From: ubuntu:20.04
        apt-get -y update
        apt-get -y install cowsay lolcat
        export LC_ALL=C
        export PATH=/usr/games:$PATH
        date | cowsay | lolcat

    Change current directory to local filesystem to build container, then move container to your cluster home to run:

    cd /localscratch/
    apptainer build --fakeroot lolcow.sif /cluster/home/USERNAME/lolcow.def
    mv lolcow.sif /cluster/home/USERNAME/
    cd /cluster/home/USERNAME/
    apptainer run lolcow.sif 
    < Fri Feb 9 10:04:24 CET 2024 >
            \   ^__^
             \  (oo)\_______
                (__)\       )\/\
                    ||----w |
                    ||     ||

    Container with old Python release

    Create definition file in your home directory python-3.7.4.def

    $ nano python-3.7.4.def
    Bootstrap: docker
    From: python:3.7.4
        pip3 install numpy
        pip3 install matplotlib

    Change directory to local storage to build container:

    cd /localscratch/
    apptainer build --fakeroot python-3.7.4.sif /cluster/home/USERNAME/python-3.7.4.def 

    Move container to your home directory, and change your current directory to home as well:

    mv python-3.7.4.sif /cluster/home/USERNAME/
    cd /cluster/home/USERNAME/

    Execute python from the container:

    apptainer exec python-3.7.4.sif python3 -V
    Python 3.7.4
    Scroll to Top