Slurm job arrays

job_array_demo.sh

#!/bin/bash
# Illustrating the use of job arrays in SLURM
# use the command: sbatch --array=1-100 job_array_demo.sh

# Partition for the job:
#SBATCH -p main

# Multithreaded (SMP) job: must run on one node
#SBATCH --nodes=1

# The name of the job:
#SBATCH --job-name="job_arrays"

# The project ID which this job should run under:
# This may or may not be needed
#SBATCH --account="999999"

# Maximum number of tasks/CPU cores used by the job:
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1

# The amount of memory in megabytes per process in the job:
#SBATCH --mem=200

# Send yourself an email when the job:
# aborts abnormally (fails)
#SBATCH --mail-type=FAIL
# begins
#SBATCH --mail-type=BEGIN
# ends successfully
#SBATCH --mail-type=END

# Use this email address:
#SBATCH --mail-user=some.email@your_institutions.com

# The maximum running time of the job in days-hours:mins:sec
#SBATCH --time=0-0:00:05

# check that the script is launched with sbatch
if [ "x$SLURM_JOB_ID" == "x" ]; then
   echo "You need to submit your job to the queuing system with sbatch"
   exit 1
fi
# Get the line corresponding the to the task number
# from a file called words
WORD=$(sed -n "${SLURM_ARRAY_TASK_ID}p" words)
# Do something with it information
# Some available env variables include:
# $SLURM_ARRAY_TASK_ID -- index of the current task
# $SLURM_ARRAY_JOB_ID --- the job id in the slurm queue
# $SLURM_ARRAY_TASK_MAX --- the largest index possible for task
# $SLUM_ARRAY_TASK_MIN --- the smalles index possible for task
echo "This is task number ${SLURM_ARRAY_TASK_ID} of ${SLURM_ARRAY_TASK_MAX} from job ${SLURM_ARRAY_JOB_ID}. The dictionary word is ${WORD}."

words

1080
10-point
10th
11-point
12-point
16-point
18-point
1st
2
20-point
2,4,5-t
2,4-d
2D
2nd
30-30
3-D
3-d
3D
3M
3rd
48-point
4-D
4GL
4H
4th
5-point
5-T
5th
6-point
6th
7-point
7th
8-point
8th
9-point
9th
-a
A
A.
a
a'
a-
a.
A-1
A1
a1
A4
A5
AA
aa
A.A.A.
AAA
aaa
AAAA
AAAAAA
AAAL
AAAS
Aaberg
Aachen
AAE
AAEE
AAF
AAG
aah
aahed
aahing
aahs
AAII
aal
Aalborg
Aalesund
aalii
aaliis
aals
Aalst
Aalto
AAM
aam
AAMSI
Aandahl
A-and-R
Aani
AAO
AAP
AAPSS
Aaqbiye
Aar
Aara
Aarau
AARC
aardvark
aardvarks
aardwolf
aardwolves
Aaren
Aargau
aargh
Aarhus
Aarika
Aaron
aaron
Aaronic
aaronic
Aaronical
Aaronite
Aaronitic
Aaron's-beard
Aaronsburg
Aaronson
AARP
aarrgh
aarrghh
Aaru
AAS
aas
A'asia
aasvogel
aasvogels
AAU
AAUP
AAUW
AAVSO
AAX
A-axes
A-axis
A.B.
AB
Ab
ab
ab-
A.B.A.
ABA
Aba
aba
Ababa
Ababdeh
Ababua
abac
abaca
abacas
abacate
abacaxi
abacay
abaci
abacinate
abacination
abacisci
abaciscus
abacist
aback
abacli
Abaco
abacot
abacterial
abactinal
abactinally
abaction
abactor
abaculi
abaculus
abacus
abacuses
Abad
abada
Abadan
Abaddon
abaddon
abadejo
abadengo
abadia
Abadite
abaff
abaft
Abagael
Abagail
Abagtha
Abailard
abaisance
abaised
abaiser
abaisse
abaissed
abaka
Abakan
abakas
Abakumov
abalation
abalienate
abalienated
abalienating
abalienation
abalone
abalones
Abama
abamp
abampere
abamperes
abamps
Abana
aband

INSTRUCTIONS

To try this out, copy the first file to your HPC system, and call it job_array_demo.sh. Copy the other file, and call it words. Usually, HPC systems have a scratch folder for such things. Make a folder in your scratch.

Then, run sbatch --array=1-100 job_array_demo.sh. That is all!

Modify for your own needs. Read more about Slurm Job Arrays here: https://slurm.schedmd.com/job_array.html.

How you call the array will determine the indices. You can have, for instance:
sbatch --array 1-7:2 job_array_demo.sh to output only lines 1, 3, 5, and 7 of the words file.

Enjoy!