Underworld running with Singularity - Gadi (runner.sh)

README_SINGULARITY.md

HPC commands to download and run the Underworld sigularity container.

REMEMBER to change the following strings:

• /FOO/BAR/ to your desired path, i.e gdata or scratch.
• IMAGE-NAME.sif to what you want the container file to be called.

These two strings MUST match up with the same strings in runner scripts.

For Gadi gadi.nci.org.au - Info on gadi.

module load singularity
singularity pull --dir /FOO/BAR/ IMAGE-NAME.sif docker://underworldcode/underworld2:2.16.4-ompi

For Setonix setonix.pawsey.org.au - Info on setonix.

module load singularity
singularity pull --dir /FOO/BAR/ IMAGE-NAME.sif docker://underworldcode/underworld2:2.16.4-mpich

To launch a singularity model

Once the .sif image is downloaded use one of the following runner.sh scripts below. (use the correct script for the machine you're on). Ensure you reference the correct path to the .sif file in the runner.sh. The model.py file should be in the same directory as the runner.sh.

For Gadi

qsub gadi_runner.sh

For Setonix

qsub setonix_runner.sh

---

Gadi specific information can be found here

gadi_runner.sh

#!/bin/bash -l

#PBS -P m18
#PBS -q normal
#PBS -l ncpus=3
#PBS -l walltime=04:00:00
#PBS -l mem=30Gb
#PBS -l jobfs=2Gb
#PBS -l software=underworld
#PBS -l wd

# load Singularity
module load singularity 
module load openmpi/4.1.4
 
export singularityDir=/FOO/BAR

# Define the container to use
export containerImage=$singularityDir/IMAGE-NAME.sif
export model="modelName.py"

# execute
mpiexec -n $PBS_NCPUS singularity exec -B /opt/pbs/default/lib/,/half-root/ --bind /lib64:/glib64 \
   $containerImage bash -c "LD_LIBRARY_PATH=/apps/openmpi/4.1.4/lib:/glib64:$LD_LIBRARY_PATH python3 $model"

setonix_runner.sh

#!/bin/bash -l
 
#SBATCH --account=projectName
#SBATCH --job-name=uw216
#SBATCH --ntasks=3
#SBATCH --ntasks-per-node=24
#SBATCH --time=00:10:00
#SBATCH --export=none
 
# load Singularity
module load singularity/3.8.6
 
export singularityDir=/FOO/BAR

# Define the container to use
export containerImage=$singularityDir/IMAGE-NAME.sif
export model="modelName.py"

# as per
# https://support.pawsey.org.au/documentation/pages/viewpage.action?pageId=116131367#UsewithSingularity-RunningPythonandR
# we unset all the host python-related ENV vars
unset $( env | grep ^PYTHON | cut -d = -f 1 | xargs )

# execute
srun --export=all -u -n $SLURM_NTASKS singularity exec -B ${PWD}:/work $containerImage bash -c "cd /work/; python3 $model"