ionox0

Here are all of the current arguments related to input / output / temporary directories used by Toil and cwltool:

---

cwltool:

--outdir - Final outputs directory

--tmpdir-prefix - This is a prefix for folders that do not yet exist, but will be used as intermediate working directories

ionox0

# Uses bsub with rsync to copy large folders
#
# Uses a walltime limit of 60 hours (should be enough for very large folders)
#
# Runs in parallel across each subfolder of the source folder, and creates logs for each as well
#
# Usage: python bsub_copy.py /source/folder /dest/folder

import os
import sys

ionox0

Building off of the CWL Recommended Best Practices

1. Use label: to describe what happens in steps:

class: Workflow

label: A workflow to do 1. 2. 3.

steps:

ionox0

# Todo: you must already be in the directory that you are trying to copy

output_location = '/some/other/place'

for (root, dirs, files) in os.walk('.'):
  # Make the new directory
  if not os.path.exists(os.path.join(output_location, root)):
    os.mkdir(os.path.join(output_location, root))
   # Make the new (empty) files
  for file in files:

ionox0

Run the conformance tests using toil, specifically for the lsf batch system

./run_test.sh RUNNER=toil-cwl-runner EXTRA="--batchSystem lsf"

ionox0

Thursday - Intros, topics

CWL 1.1
- WallTime
- 

Extensions
- 

runIf --> when

ionox0

In case anyone’s still wondering how Toil’s lsf.py job tracking works on Juno.

bjobs -l <jobid> will work on Juno, but the string that is attempting to be matched is wrong:

Tue Sep 24 19:12:37: Started 1 Task(s) on Host(s) <jx15>, Allocated 1 Slot(s) 

DOES NOT MATCH WITH:

                elif line.find("Started on ") > -1:

ionox0

For comprehensive Toil explanation see: https://media.nature.com/original/nature-assets/nbt/journal/v35/n4/extref/nbt.3772-S1.pdf

See logs for just one job (by using the full log file, requires knowing the job's toil-generated ID)

cat cwltoil.log | grep jobVM1fIs

Grep for full commands from toil logs

This gives you a more concise view of the commands being run (note that this information is only available from Toil when running with --logDebug, or when using the pipeline_submit script with --logLevel DEBUG)

ionox0

Note: The "job" that we're dealing with in this case is actually an instance of the Toil JobGraph class. Here is the JobGraph class hierarchy. You can see that a JobGraph is similar to a Job, as it is one of it's descendants.

Here is the job class for completeness's sake.

Let's try an example

From the Toil log files, you will be able to find the Toil job ID for the failed job. Here we show a job being submitted for the third time after two failed attempts. We can see the cwl file that defines the job, along with resource requirements, and the bsub command that is issued.

1.

There is then logging to indicate the failure of the job with job ID 9ntS9h.

ionox0

From the Toil log files, you will be able to find the Toil job ID for the failed job. Here we show a job being submitted for the third time after two failed attempts. We can see the cwl file that defines the job, along with resource requirements, and the bsub command that is issued.

1.

There is then logging to indicate the failure of the job with job ID 9ntS9h.

DEBUG:toil.batchSystems.abstractGridEngineBatchSystem:Issued the job command: /home/johnsoni/virtualenvs/pipeline_1.1.14/bin/_toil_worker file:///home/johnsoni/pipeline_1.1.14/ACCESS-Pipeline/cwl_tools/trimgalore/trimgalore.cwl file:/home/johnsoni/juno_ACCESS/5500-FZ/5500-FZ-1.1.14/tmp/jobstore-2888ebd4-bd4a-11e9-a01d-ec0d9a88a15a u/v/job9ntS9h with job id: 45
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_1.1.14/ACCESS-Pipeline/cwl_tools/trimgalore/trimgalore.cwl' u/v/job9ntS9h with job batch system ID: 45 and cores: 2, disk: 20.5 G, and memory: 15.6 G

...