Submitting multiple R jobs to bluecrystal 3 and 4

rscript.r

# Get the list of arguments passed to R
args <- commandArgs(T)

# Get the first argument, make it numeric because by default it is read as a string
job_id <- as.numeric(args[1])

# Print the argument
message("job number ", job_id)

submit_bc3.sh

#!/bin/bash

#PBS -N some_job_name
#PBS -o job_reports/meta_16-output
#PBS -e job_reports/meta_16-error
#PBS -t 1-10
#PBS -l walltime=00:15:00
#PBS -l nodes=1:ppn=1
#PBS -S /bin/bash

module add languages/r/3.4.4

echo "Running on ${HOSTNAME}"

if [ -n "${1}" ]; then
  echo "${1}"
  PBS_ARRAYID=${1}
fi

i=${PBS_ARRAYID}

# Run the r script, passing the job ID

Rscript rscript.r ${i}

submit_bc4.sh

#!/bin/bash

#SBATCH --job-name=some_job_name
#SBATCH --nodes=1
#SBATCH --mem=4G
#SBATCH --ntasks=1
#SBATCH --time=0-00:15:00
#SBATCH --array=1-10
#SBATCH --partition=veryshort
#SBATCH --output=job_reports/slurm-%A_%a.out

module add languages/r/3.4.4

echo "Running on ${HOSTNAME}"

if [ -n "${1}" ]; then
  echo "${1}"
  SLURM_ARRAY_TASK_ID=${1}
fi

i=${SLURM_ARRAY_TASK_ID}


# Run the r script, passing the job ID

Rscript rscript.r ${i}

Quick example of how to parallelise an R script. In this simple example we want to print the values 1-10 in parallel. To run in bc3:

The submission script has a bunch of flags, but the key is the -t (bc3) or --array (bc4) flag. This says 'run this same script 10 times, and for each run set a variable to the job number, from 1 to 10'. The variable is set as PBS_ARRAYID in bc3 or SLURM_ARRAY_TASK_ID in bc4.

qsub submit_bc3.sh

Or to test e.g. for job number 3

source submit_bc3.sh 3

This should just print out the "job number 3". Similarly on bc4:

sbatch submit_bc4.sh

or to test in the front end:

source submit_bc4.sh 3