Modules
Quick video introduction:
Software on IDUN is installed using the Easybuild software installation tool. Easybuild is configured to use the Lmod modules tool to change the environment via modulefiles. The module naming scheme is chosen to give a hierarchically organized modules setup. This means when logging into IDUN listing the available modules will show the 'Core' modules including the compilers and in addition software packages not dependent on any other installed software (e.g. Matlab):
$ module available ---------------------------------------- /share/apps/modules/all/Core ----------------------------------------- FLUENT/18.0 GCC/5.4.0-2.26 (L) Java/1.8.0_92 foss/2016a foss/2017a (D) GCC/4.9.3-2.25 GCC/6.3.0-2.27 (D) MATLAB/2016b foss/2016b
Default modules, i.e. modules that will be loaded if no version number is specified, is marked (D). To see software built with a specific compiler, load the compiler module and again list available modules, e.g.:
$ module load GCC/5.4.0-2.26 $ module available ------------------------------ /share/apps/modules/all/Compiler/GCC/5.4.0-2.26 -------------------------------- CMake/3.5.2 (D) GSL/2.3 METIS/5.1.0 OpenBLAS/0.2.18-LAPACK-3.6.1 OpenMPI/1.10.3 (L) ---------------------------------------- /share/apps/modules/all/Core ----------------------------------------- FLUENT/18.0 GCC/5.4.0-2.26 (L) Java/1.8.0_92 foss/2016a foss/2017a (D) GCC/4.9.3-2.25 GCC/6.3.0-2.27 (D) MATLAB/2016b foss/2016b
Loaded modules are marked (L). As a third step, load the OpenMPI module to see software built with the specific compiler and MPI versions:
$ module load OpenMPI/1.10.3 $ module available -------------------------- /share/apps/modules/all/MPI/GCC/5.4.0-2.26/OpenMPI/1.10.3 -------------------------- Boost/1.63.0-Python-2.7.12 HDF5/1.8.17 ParMETIS/4.0.3 ScaLAPACK/2.0.2-OpenBLAS-0.2.18-LAPACK-3.6.1 FFTW/3.3.4 Hypre/2.11.1 Python/2.7.12 h5py/2.6.0-Python-2.7.12-HDF5-1.8.17 GDAL/2.1.0 PROJ/4.9.2 R/3.3.1 etc. ------------------------------ /share/apps/modules/all/Compiler/GCC/5.4.0-2.26 -------------------------------- CMake/3.5.2 (D) GSL/2.3 METIS/5.1.0 OpenBLAS/0.2.18-LAPACK-3.6.1 OpenMPI/1.10.3 (L) ---------------------------------------- /share/apps/modules/all/Core ----------------------------------------- FLUENT/18.0 GCC/5.4.0-2.26 (L) Java/1.8.0_92 foss/2016a foss/2017a (D) GCC/4.9.3-2.25 GCC/6.3.0-2.27 (D) MATLAB/2016b foss/2016b
EasyBuild Toolchains
As an alternative to the above chain of 'module load' commands one can load a toolchain module listed under the 'Core' modules directly. A toolchain is a bundle of a compiler and frequently used libraries which is needed for building a software package/application. The toolchains installed on Idun include the so-called common toolchains foss (short for "Free and Open Source Software") and intel, and in addition the iomkl toolchain. The available toolchains on Idun with the components that comprise the toolchain is listed below
| Toolchain | Compilers | MPI stack | Included libraries |
|---|---|---|---|
| foss | GCC | OpenMPI | OpenBLAS/LAPACK, ScaLAPACK, FFTW |
| intel | icc, ifort | Intel MPI | Intel MKL |
| iomkl | icc, ifort | OpenMPI | Intel MKL |
E.g. loading the foss/2016b toolchain module directly and issuing the module available command will show the output listed above with modules loaded marked with (L).
Using the module spider command
To see a full list of available modules installed on the system enter command
$ module spider
and adding a specific module file will show detailed information about that package, e.g. what module files to load to use the software, e.g.
$ module spider HDF5/1.8.17
...
You will need to load all module(s) on any one of the lines below before the "netCDF/4.4.1" module is available to load.
GCC/5.4.0-2.26 OpenMPI/1.10.3
...
Further Information
Creating Python virtual environments
With the use of 'virtualenv' a user can create his/her own virtual Python environment based on a loaded Python version:
$ module load intel/2018b
$ module load Python/3.6.6
$ module list
Currently Loaded Modules:
1) GCCcore/.7.3.0 5) impi/2018.3.222 9) zlib/.1.2.11 13) GMP/.6.1.2
2) binutils/.2.30 6) imkl/2018.3.222 10) ncurses/.6.1 14) libffi/.3.2.1
3) icc/2018.3.222-GCC-7.3.0-2.30 7) intel/2018b 11) libreadline/.7.0 15) Python/3.6.6
4) ifort/2018.3.222-GCC-7.3.0-2.30 8) bzip2/.1.0.6 12) XZ/.5.2.4
$ type virtualenv
virtualenv is /share/apps/software/MPI/intel/2018.3.222-GCC-7.3.0-2.30/impi/2018.3.222/Python/3.6.6/bin/virtualenv
$ virtualenv datasci
Using base prefix '/share/apps/software/MPI/intel/2018.3.222-GCC-7.3.0-2.30/impi/2018.3.222/Python/3.6.6'
New python executable in /lustre1/work/bjornlin/pythonenvs/datasci/bin/python
Installing setuptools, pip, wheel...done.
$ ls
datasci
$ source datasci/bin/activate
(datasci)$ pip install scipy numpy scikit-learn pandas matplotlib
# To exit at virtual environment do deactivate
(datasci)$ deactivate