Gadi uw3 workflow

gadi.md

Guide to running Underworld3 on Gadi, baremetal only for now.

Log in and make sure your modulepath is good for Underworld custom modulefiles.

1. export MODULEPATH=/g/data/m18/modulefiles/:$MODULEPATH should be in your .bashrc.

2. module load underworld/dev_env - this DOESN"T load underworld3 but sets up an environment to compile and run the code on Gadi.

3. Pull down the code and compile Using pip directly to install the development version.

pip install --no-cache-dir git+https://github.com/underworldcode/underworld3@development --user

Using git to clone the repository and build it with pip (better for development on Gadi)

git clone https://github.com/underworldcode/underworld3.git
cd underworld3
pip install -e .

3. Run the code. First create a pbs script like this, let's call it runner.pbs

#!/bin/bash

#PBS -P m18
#PBS -q normal
#PBS -l walltime=00:05:00
#PBS -l mem=2GB
#PBS -l jobfs=1GB
#PBS -l ncpus=1
#PBS -l software=underworld
#PBS -l wd

#PBS -l storage=gdata/m18

export MODULEPATH=/g/data/m18/modulefiles:$MODULEPATH
module load underworld/dev_env

# disable numpy intrinsic parallelism
export OPENBLAS_NUM_THREADS=1

# print out environment
env

# execution
mpiexec python3 input.py

Then execute qsub runner.pbs

And, before that, export MODULEPATH=/g/data/m18/modulefiles:$MODULEPATH in order to load the module

Thanks for the points - updates made.