Guide to install and use `toil-cwl-runner` on Snellius

cwl-runner

#!/bin/bash

# Convenience for executing Common Workflow Language workflows on the Snellius supercomputer
# using toil-cwl-runner from http://toil.ucsc-cgl.org/

# This script accepts all toil-cwl-runner options:
# https://toil.readthedocs.io/en/latest/running/cwl.html#detailed-usage-instructions
# Example usage:
#   cwl-runner my_workflow.cwl my_inputs.yaml
# will read the CWL workflow description "my_workflow.cwl" using the inputs specified
# in "my_inputs.yaml" and submit jobs to Slurm.

default_queue="--partition=thin"
base_options="--batchSystem slurm --disableCaching true --singularity" 

# Some Snellius-specific defaults are set, as shown above.

# To specify a different queue and/or to set additional slurm options
# set TOIL_SLURM_ARGS

# Example:
#   TOIL_SLURM_ARGS="--partition=gpu --gpus=1" cwl-runner my_gpu_workflow.cwl my_inputs.yml

export TOIL_SLURM_ARGS="${TOIL_SLURM_ARGS:-${default_queue}} --export=ALL"
# must have `--export=all` for toil-cwl-runner to work
export PYTHONIOENCODING=utf8
# Ensures that unicode (including emojis🎉) is properly represented on the command line

echo "cwl-runner@snellius using toil-cwl-runner. Base options: \"${base_options}\" TOIL_SURM_ARGS=\"${TOIL_SLURM_ARGS}\""

toil-cwl-runner ${base_options} $@

snellius-toil-cwl-runner-guide.md

Guide to install toil-cwl-runner on Snellius

https://toil.readthedocs.io/en/latest/running/cwl.html

https://servicedesk.surfsara.nl/wiki/display/WIKI/Snellius

module load 2021 nodejs/14.17.0-GCCcore-10.3.0
# good to add the above to your ~/.bashrc
# this speeds up processing of CWL Expressions and CWL Parameter references

export TOIL_SLURM_ARGS="--partition=thin --export=ALL"
# sets the partition to use; must have `--export=all` for toil-cwl-runner to work
# good to add the above to your ~/.bashrc as well

export PYTHONIOENCODING=utf8
# Ensures that unicode (including emojis🎉) is properly represented on the command line
# good to add the above to your ~/.bashrc as well


python3 -m venv toilenv
source toilenv/bin/activate
pip install -U pip setuptools wheel
pip install toil[cwl]

After the above is done, use the following to parse and submit your workflow

toil-cwl-runner --batchSystem slurm --disableCaching --singularity path_to_your_workflow.cwl path_to_workflow_inputs.json

To run the CWL v1.2 conformance tests

pip install cwltest
git clone https://github.com/common-workflow-language/cwl-v1.2.git
cd cwl-v1.2
TMPDIR=$PWD  ./run_test.sh RUNNER=toil-cwl-runner EXTRA="--batchSystem slurm --disableCaching --singularity"
# any extra option we would want to pass to `toil-cwl-runner` goes in the `EXTRA=` string

# optionally add `-j16` or similar to control the number of simultaneous tests being run
# TMPDIR=$PWD  ./run_test.sh RUNNER=toil-cwl-runner EXTRA="--batchSystem slurm --disableCaching --singularity " -j16
# and/or optionally add `--junit-xml path.xml` to save details about the results
# TMPDIR=$PWD  ./run_test.sh RUNNER=toil-cwl-runner EXTRA="--batchSystem slurm --disableCaching --singularity " -j16 --junit-xml $HOME/toil-5.5.0_cwl_v1_2.xml

If you are using InplaceUpdateRequirement in your CWL v1.1 or CWL v1.2+ description, then you'll need to also include --bypass-file-store to your toil-cwl-runner invocation.

Results on 2021-11-08 with Toil 5.5.0 and --bypass-file-store on Snellius using Singularity 3.8.3-1.el8 :

337 tests passed, 1 failures, 0 unsupported features

The one failure is due to known a limitation of Singularity with respect to Docker format containers that use ENTRYPOINT (which is not recommended anyhow)