toil-cwl-bcbio-20170301.log

 * AWS

e1d21297-981b-4070-8c53-5412d27796eb    INFO:toil:Running Toil version 3.7.0a1.dev356-7bb59638ab5ac9b38a66f0f703a334ab684f4d42.
e1d21297-981b-4070-8c53-5412d27796eb    INFO:toil.fileStore:Starting job ('file:///home/run/test_bcbio_cwl-master/run_info-cwl-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl e1d21297-981b-4070-8c53-5412d27796eb) with ID (b53cfcd7c9f47ecf71a3ff12be4464ea1d121fac).
e1d21297-981b-4070-8c53-5412d27796eb    [job prep_samples.cwl] /tmp/tmp5SUv3u$ docker \
e1d21297-981b-4070-8c53-5412d27796eb        run \
e1d21297-981b-4070-8c53-5412d27796eb        -i \
e1d21297-981b-4070-8c53-5412d27796eb        --volume=/var/lib/mesos/slaves/d5be82f5-b509-48c4-a2cb-5873a775069e-S1/frameworks/d5be82f5-b509-48c4-a2cb-5873a775069e-0001/executors/toil-199/runs/26162efd-60e2-4948-b100-e6a35284c261/toil-f6972eb1-c54c-4430-8fc0-ad4fe85c34b1/tmpdm7zpb/c072ed45-2da0-4f8c-bf45-d2eaca6e0608/t3kGuVh/inp/tmpI0E9a4/hg19.fa.fai:/var/lib/cwl/stg6b98e09d-d990-45bc-8bd6-706b83197bbc/hg19.fa.fai:ro \
e1d21297-981b-4070-8c53-5412d27796eb        --volume=/var/lib/mesos/slaves/d5be82f5-b509-48c4-a2cb-5873a775069e-S1/frameworks/d5be82f5-b509-48c4-a2cb-5873a775069e-0001/executors/toil-199/runs/26162efd-60e2-4948-b100-e6a35284c261/toil-f6972eb1-c54c-4430-8fc0-ad4fe85c34b1/tmpdm7zpb/c072ed45-2da0-4f8c-bf45-d2eaca6e0608/t3kGuVh/inp/tmpI0E9a4/hg19.dict:/var/lib/cwl/stg6b98e09d-d990-45bc-8bd6-706b83197bbc/hg19.dict:ro \
e1d21297-981b-4070-8c53-5412d27796eb        --volume=/var/lib/mesos/slaves/d5be82f5-b509-48c4-a2cb-5873a775069e-S1/frameworks/d5be82f5-b509-48c4-a2cb-5873a775069e-0001/executors/toil-199/runs/26162efd-60e2-4948-b100-e6a35284c261/toil-f6972eb1-c54c-4430-8fc0-ad4fe85c34b1/tmpdm7zpb/c072ed45-2da0-4f8c-bf45-d2eaca6e0608/t3kGuVh/inp/tmpI0E9a4/hg19.fa:/var/lib/cwl/stg6b98e09d-d990-45bc-8bd6-706b83197bbc/hg19.fa:ro \
e1d21297-981b-4070-8c53-5412d27796eb        --volume=/var/lib/mesos/slaves/d5be82f5-b509-48c4-a2cb-5873a775069e-S1/frameworks/d5be82f5-b509-48c4-a2cb-5873a775069e-0001/executors/toil-199/runs/26162efd-60e2-4948-b100-e6a35284c261/toil-f6972eb1-c54c-4430-8fc0-ad4fe85c34b1/tmpdm7zpb/c072ed45-2da0-4f8c-bf45-d2eaca6e0608/t3kGuVh/inp/tmpqGLDkW/variant_regions-bam.bed:/var/lib/cwl/stg49edbfa7-366f-47c8-8b1b-c59f0627df4c/variant_regions-bam.bed:ro \
e1d21297-981b-4070-8c53-5412d27796eb        --volume=/tmp/tmp5SUv3u:/var/spool/cwl:rw \
e1d21297-981b-4070-8c53-5412d27796eb        --volume=/var/lib/mesos/slaves/d5be82f5-b509-48c4-a2cb-5873a775069e-S1/frameworks/d5be82f5-b509-48c4-a2cb-5873a775069e-0001/executors/toil-199/runs/26162efd-60e2-4948-b100-e6a35284c261/toil-f6972eb1-c54c-4430-8fc0-ad4fe85c34b1/tmpdm7zpb/c072ed45-2da0-4f8c-bf45-d2eaca6e0608/tCrTkMZ/tmp:/tmp:rw \
e1d21297-981b-4070-8c53-5412d27796eb        --workdir=/var/spool/cwl \
e1d21297-981b-4070-8c53-5412d27796eb        --read-only=true \
e1d21297-981b-4070-8c53-5412d27796eb        --rm \
e1d21297-981b-4070-8c53-5412d27796eb        --env=TMPDIR=/tmp \
e1d21297-981b-4070-8c53-5412d27796eb        --env=HOME=/var/spool/cwl \
e1d21297-981b-4070-8c53-5412d27796eb        --env=MPLCONFIGDIR=. \
e1d21297-981b-4070-8c53-5412d27796eb        bcbio/bcbio \
e1d21297-981b-4070-8c53-5412d27796eb        bcbio_nextgen.py \
e1d21297-981b-4070-8c53-5412d27796eb        runfn \
e1d21297-981b-4070-8c53-5412d27796eb        prep_samples \
e1d21297-981b-4070-8c53-5412d27796eb        cwl \
e1d21297-981b-4070-8c53-5412d27796eb        sentinel-runtime=cores,2,ram,4096 \
e1d21297-981b-4070-8c53-5412d27796eb        sentinel-parallel=multi-parallel \
e1d21297-981b-4070-8c53-5412d27796eb        sentinel-outputs=config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready \
e1d21297-981b-4070-8c53-5412d27796eb        config__algorithm__variant_regions=/var/lib/cwl/stg49edbfa7-366f-47c8-8b1b-c59f0627df4c/variant_regions-bam.bed \
e1d21297-981b-4070-8c53-5412d27796eb        reference__fasta__base=/var/lib/cwl/stg6b98e09d-d990-45bc-8bd6-706b83197bbc/hg19.fa \
e1d21297-981b-4070-8c53-5412d27796eb        description=Test2
e1d21297-981b-4070-8c53-5412d27796eb    INFO:cwltool:[job prep_samples.cwl] /tmp/tmp5SUv3u$ docker \
e1d21297-981b-4070-8c53-5412d27796eb        run \
e1d21297-981b-4070-8c53-5412d27796eb        -i \
e1d21297-981b-4070-8c53-5412d27796eb        --volume=/var/lib/mesos/slaves/d5be82f5-b509-48c4-a2cb-5873a775069e-S1/frameworks/d5be82f5-b509-48c4-a2cb-5873a775069e-0001/executors/toil-199/runs/26162efd-60e2-4948-b100-e6a35284c261/toil-f6972eb1-c54c-4430-8fc0-ad4fe85c34b1/tmpdm7zpb/c072ed45-2da0-4f8c-bf45-d2eaca6e0608/t3kGuVh/inp/tmpI0E9a4/hg19.fa.fai:/var/lib/cwl/stg6b98e09d-d990-45bc-8bd6-706b83197bbc/hg19.fa.fai:ro \
e1d21297-981b-4070-8c53-5412d27796eb        --volume=/var/lib/mesos/slaves/d5be82f5-b509-48c4-a2cb-5873a775069e-S1/frameworks/d5be82f5-b509-48c4-a2cb-5873a775069e-0001/executors/toil-199/runs/26162efd-60e2-4948-b100-e6a35284c261/toil-f6972eb1-c54c-4430-8fc0-ad4fe85c34b1/tmpdm7zpb/c072ed45-2da0-4f8c-bf45-d2eaca6e0608/t3kGuVh/inp/tmpI0E9a4/hg19.dict:/var/lib/cwl/stg6b98e09d-d990-45bc-8bd6-706b83197bbc/hg19.dict:ro \
e1d21297-981b-4070-8c53-5412d27796eb        --volume=/var/lib/mesos/slaves/d5be82f5-b509-48c4-a2cb-5873a775069e-S1/frameworks/d5be82f5-b509-48c4-a2cb-5873a775069e-0001/executors/toil-199/runs/26162efd-60e2-4948-b100-e6a35284c261/toil-f6972eb1-c54c-4430-8fc0-ad4fe85c34b1/tmpdm7zpb/c072ed45-2da0-4f8c-bf45-d2eaca6e0608/t3kGuVh/inp/tmpI0E9a4/hg19.fa:/var/lib/cwl/stg6b98e09d-d990-45bc-8bd6-706b83197bbc/hg19.fa:ro \
e1d21297-981b-4070-8c53-5412d27796eb        --volume=/var/lib/mesos/slaves/d5be82f5-b509-48c4-a2cb-5873a775069e-S1/frameworks/d5be82f5-b509-48c4-a2cb-5873a775069e-0001/executors/toil-199/runs/26162efd-60e2-4948-b100-e6a35284c261/toil-f6972eb1-c54c-4430-8fc0-ad4fe85c34b1/tmpdm7zpb/c072ed45-2da0-4f8c-bf45-d2eaca6e0608/t3kGuVh/inp/tmpqGLDkW/variant_regions-bam.bed:/var/lib/cwl/stg49edbfa7-366f-47c8-8b1b-c59f0627df4c/variant_regions-bam.bed:ro \
e1d21297-981b-4070-8c53-5412d27796eb        --volume=/tmp/tmp5SUv3u:/var/spool/cwl:rw \
e1d21297-981b-4070-8c53-5412d27796eb        --volume=/var/lib/mesos/slaves/d5be82f5-b509-48c4-a2cb-5873a775069e-S1/frameworks/d5be82f5-b509-48c4-a2cb-5873a775069e-0001/executors/toil-199/runs/26162efd-60e2-4948-b100-e6a35284c261/toil-f6972eb1-c54c-4430-8fc0-ad4fe85c34b1/tmpdm7zpb/c072ed45-2da0-4f8c-bf45-d2eaca6e0608/tCrTkMZ/tmp:/tmp:rw \
e1d21297-981b-4070-8c53-5412d27796eb        --workdir=/var/spool/cwl \
e1d21297-981b-4070-8c53-5412d27796eb        --read-only=true \
e1d21297-981b-4070-8c53-5412d27796eb        --rm \
e1d21297-981b-4070-8c53-5412d27796eb        --env=TMPDIR=/tmp \
e1d21297-981b-4070-8c53-5412d27796eb        --env=HOME=/var/spool/cwl \
e1d21297-981b-4070-8c53-5412d27796eb        --env=MPLCONFIGDIR=. \
e1d21297-981b-4070-8c53-5412d27796eb        bcbio/bcbio \
e1d21297-981b-4070-8c53-5412d27796eb        bcbio_nextgen.py \
e1d21297-981b-4070-8c53-5412d27796eb        runfn \
e1d21297-981b-4070-8c53-5412d27796eb        prep_samples \
e1d21297-981b-4070-8c53-5412d27796eb        cwl \
e1d21297-981b-4070-8c53-5412d27796eb        sentinel-runtime=cores,2,ram,4096 \
e1d21297-981b-4070-8c53-5412d27796eb        sentinel-parallel=multi-parallel \
e1d21297-981b-4070-8c53-5412d27796eb        sentinel-outputs=config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready \
e1d21297-981b-4070-8c53-5412d27796eb        config__algorithm__variant_regions=/var/lib/cwl/stg49edbfa7-366f-47c8-8b1b-c59f0627df4c/variant_regions-bam.bed \
e1d21297-981b-4070-8c53-5412d27796eb        reference__fasta__base=/var/lib/cwl/stg6b98e09d-d990-45bc-8bd6-706b83197bbc/hg19.fa \
e1d21297-981b-4070-8c53-5412d27796eb        description=Test2
e1d21297-981b-4070-8c53-5412d27796eb    [2017-02-24T19:09Z] 2e33e7e12214: Prepare cleaned BED file : Test2
e1d21297-981b-4070-8c53-5412d27796eb    [2017-02-24T19:09Z] 2e33e7e12214: bgzip variant_regions-bam.bed
e1d21297-981b-4070-8c53-5412d27796eb    [2017-02-24T19:09Z] 2e33e7e12214: tabix index variant_regions-bam.bed.gz
e1d21297-981b-4070-8c53-5412d27796eb    [2017-02-24T19:09Z] 2e33e7e12214: Prepare merged BED file : Test2
e1d21297-981b-4070-8c53-5412d27796eb    [2017-02-24T19:09Z] 2e33e7e12214: bgzip variant_regions-bam-merged.bed
e1d21297-981b-4070-8c53-5412d27796eb    [2017-02-24T19:09Z] 2e33e7e12214: tabix index variant_regions-bam-merged.bed.gz
e1d21297-981b-4070-8c53-5412d27796eb    [job prep_samples.cwl] Job error:
e1d21297-981b-4070-8c53-5412d27796eb    Error validating output record, * the `config__algorithm__variant_regions` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        is not a dict
e1d21297-981b-4070-8c53-5412d27796eb    * the `config__algorithm__variant_regions_merged` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        value is a list, expected File or null
e1d21297-981b-4070-8c53-5412d27796eb    * the `config__algorithm__variant_regions_orig` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        value is a list, expected File or null
e1d21297-981b-4070-8c53-5412d27796eb    * the `config__algorithm__coverage` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        value is a list, expected File or null
e1d21297-981b-4070-8c53-5412d27796eb    * the `config__algorithm__coverage_merged` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        value is a list, expected File or null
e1d21297-981b-4070-8c53-5412d27796eb    * the `config__algorithm__coverage_orig` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        value is a list, expected File or null
e1d21297-981b-4070-8c53-5412d27796eb    * the `config__algorithm__seq2c_bed_ready` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        value is a list, expected File or null
e1d21297-981b-4070-8c53-5412d27796eb     in {
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__coverage_orig": [],
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__coverage_merged": [],
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__variant_regions_merged": [],
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__variant_regions": [],
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__variant_regions_orig": [],
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__coverage": [],
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__seq2c_bed_ready": []
e1d21297-981b-4070-8c53-5412d27796eb    }
e1d21297-981b-4070-8c53-5412d27796eb    ERROR:cwltool:[job prep_samples.cwl] Job error:
e1d21297-981b-4070-8c53-5412d27796eb    Error validating output record, * the `config__algorithm__variant_regions` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        is not a dict
e1d21297-981b-4070-8c53-5412d27796eb    * the `config__algorithm__variant_regions_merged` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        value is a list, expected File or null
e1d21297-981b-4070-8c53-5412d27796eb    * the `config__algorithm__variant_regions_orig` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        value is a list, expected File or null
e1d21297-981b-4070-8c53-5412d27796eb    * the `config__algorithm__coverage` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        value is a list, expected File or null
e1d21297-981b-4070-8c53-5412d27796eb    * the `config__algorithm__coverage_merged` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        value is a list, expected File or null
e1d21297-981b-4070-8c53-5412d27796eb    * the `config__algorithm__coverage_orig` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        value is a list, expected File or null
e1d21297-981b-4070-8c53-5412d27796eb    * the `config__algorithm__seq2c_bed_ready` field is not valid because
e1d21297-981b-4070-8c53-5412d27796eb        value is a list, expected File or null
e1d21297-981b-4070-8c53-5412d27796eb     in {
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__coverage_orig": [],
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__coverage_merged": [],
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__variant_regions_merged": [],
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__variant_regions": [],
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__variant_regions_orig": [],
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__coverage": [],
e1d21297-981b-4070-8c53-5412d27796eb        "config__algorithm__seq2c_bed_ready": []
e1d21297-981b-4070-8c53-5412d27796eb    }
e1d21297-981b-4070-8c53-5412d27796eb    [job prep_samples.cwl] completed permanentFail
e1d21297-981b-4070-8c53-5412d27796eb    WARNING:cwltool:[job prep_samples.cwl] completed permanentFail
e1d21297-981b-4070-8c53-5412d27796eb    Traceback (most recent call last):
e1d21297-981b-4070-8c53-5412d27796eb      File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 340, in main
e1d21297-981b-4070-8c53-5412d27796eb        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
e1d21297-981b-4070-8c53-5412d27796eb      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1284, in _runner
e1d21297-981b-4070-8c53-5412d27796eb        returnValues = self._run(jobGraph, fileStore)
e1d21297-981b-4070-8c53-5412d27796eb      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1231, in _run
e1d21297-981b-4070-8c53-5412d27796eb        return self.run(fileStore)
e1d21297-981b-4070-8c53-5412d27796eb      File "/usr/local/lib/python2.7/dist-packages/toil/cwl/cwltoil.py", line 290, in run
e1d21297-981b-4070-8c53-5412d27796eb        raise cwltool.errors.WorkflowException(status)
e1d21297-981b-4070-8c53-5412d27796eb    WorkflowException: permanentFail
e1d21297-981b-4070-8c53-5412d27796eb    ERROR:toil.worker:Exiting the worker because of a failed job on host ip-10-0-0-10.ec2.internal

* Local

kunkel 2017-03-01 18:47:35,942 MainThread WARNING toil.leader: Job failed with exit value 1: 'CWLWorkflow' 78e340c2-d29c-4c67-a926-0d1dc40fef2d
kunkel 2017-03-01 18:47:36,033 MainThread WARNING toil.leader: No log file is present, despite job failing: 'CWLWorkflow' 78e340c2-d29c-4c67-a926-0d1dc40fef2d
kunkel 2017-03-01 18:47:36,033 MainThread WARNING toil.jobGraph: Due to failure we are reducing the remaining retry count of job 'CWLWorkflow' 78e340c2-d29c-4c67-a926-0d1dc40fef2d with ID 78e340c2-d29c-4c67-a926-0d1dc40fef2d to 0
kunkel 2017-03-01 18:47:36,200 MainThread WARNING toil.leader: Job 'CWLWorkflow' 78e340c2-d29c-4c67-a926-0d1dc40fef2d with ID 78e340c2-d29c-4c67-a926-0d1dc40fef2d is completely failed
kunkel 2017-03-01 18:47:38,201 MainThread INFO toil.leader: No jobs left to run so exiting.
kunkel 2017-03-01 18:47:38,201 MainThread INFO toil.leader: Finished the main loop
kunkel 2017-03-01 18:47:38,201 MainThread INFO toil.serviceManager: Waiting for service manager thread to finish ...
kunkel 2017-03-01 18:47:38,276 MainThread INFO toil.serviceManager: ... finished shutting down the service manager. Took 0.0744118690491 seconds
kunkel 2017-03-01 18:47:38,276 MainThread INFO toil.statsAndLogging: Waiting for stats and logging collator thread to finish ...
kunkel 2017-03-01 18:47:38,628 MainThread INFO toil.statsAndLogging: ... finished collating stats and logs. Took 0.352509021759 seconds
kunkel 2017-03-01 18:47:38,683 MainThread INFO toil.leader: Finished toil run with 1 failed jobs
kunkel 2017-03-01 18:47:38,683 MainThread INFO toil.leader: Failed jobs at end of the run: 'CWLWorkflow' 78e340c2-d29c-4c67-a926-0d1dc40fef2d
I0301 18:47:38.730059    47 sched.cpp:1987] Asked to stop the driver
I0301 18:47:38.730165   113 sched.cpp:1187] Stopping framework '9d371604-1ebe-445e-a8e6-1bf83a06e36a-0000'
Traceback (most recent call last):
  File "/usr/local/bin/cwltoil", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/toil/cwl/cwltoil.py", line 731, in main
    outobj = toil.start(wf1)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 676, in start
    return self._runMainLoop(rootJobGraph)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 984, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/usr/local/lib/python2.7/dist-packages/toil/leader.py", line 200, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'aws:us-east-1:local-bcbio-jobstore' contains 1 failed jobs: 'CWLWorkflow' 78e340c2-d29c-4c67-a926-0d1dc40fef2d