chapmanb

 * AWS

e1d21297-981b-4070-8c53-5412d27796eb    INFO:toil:Running Toil version 3.7.0a1.dev356-7bb59638ab5ac9b38a66f0f703a334ab684f4d42.
e1d21297-981b-4070-8c53-5412d27796eb    INFO:toil.fileStore:Starting job ('file:///home/run/test_bcbio_cwl-master/run_info-cwl-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl e1d21297-981b-4070-8c53-5412d27796eb) with ID (b53cfcd7c9f47ecf71a3ff12be4464ea1d121fac).
e1d21297-981b-4070-8c53-5412d27796eb    [job prep_samples.cwl] /tmp/tmp5SUv3u$ docker \
e1d21297-981b-4070-8c53-5412d27796eb        run \
e1d21297-981b-4070-8c53-5412d27796eb        -i \
e1d21297-981b-4070-8c53-5412d27796eb        --volume=/var/lib/mesos/slaves/d5be82f5-b509-48c4-a2cb-5873a775069e-S1/frameworks/d5be82f5-b509-48c4-a2cb-5873a775069e-0001/executors/toil-199/runs/26162efd-60e2-4948-b100-e6a35284c261/toil-f6972eb1-c54c-4430-8fc0-ad4fe85c34b1/tmpdm7zpb/c072ed45-2da0-4f8c-bf45-d2eaca6e0608/t3kGuVh/inp/tmpI0E9a4/hg19.fa.fai:/var/lib/cwl/stg6b98e09d-d990-45bc-8bd6-706b83197bbc/hg19.f

chapmanb

---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.7.0a1.dev352-bb39fc499ee8ebeaef8e3b1d682bfb96f2f12a2b.
INFO:toil.fileStore:Starting job ('file:///home/run/test_bcbio_cwl-master/run_info-cwl-workflow/steps/prep_align_inputs.cwl' bcbio_nextgen.py runfn prep_align_inputs cwl 306cb9bb-3600-4f06-86f9-da54535e8199) with ID (26771e8d83ba0455c040c8712bae9653c85b565e).
Got workflow error
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cwltool/main.py", line 220, in single_job_executor
    for r in jobiter:
  File "/usr/local/lib/python2.7/dist-packages/cwltool/draft2tool.py", line 283, in job
    builder.pathmapper = self.makePathMapper(reffiles, builder.stagedir, **make_path_mapper_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/cwltool/draft2tool.py", line 170, in makePathMapper

chapmanb

diff --git a/src/toil/provisioners/aws/__init__.py b/src/toil/provisioners/aws/__init__.py
index c64e9d6..830e190 100644
--- a/src/toil/provisioners/aws/__init__.py
+++ b/src/toil/provisioners/aws/__init__.py
@@ -42,7 +42,12 @@ def _getCurrentAWSZone(spotBid=None, nodeType=None, ctx=None):
         pass
     else:
         zone = os.environ.get('TOIL_AWS_ZONE', None)
-        if spotBid:
+        if not zone and runningOnEC2():

chapmanb

import boto

conn = boto.connect_vpc()
print "access_key", conn.aws_access_key_id
for vpc in conn.get_all_vpcs(filters={'tag:Name': 'bcbio'}):
    print "vpc"
    print vpc.id
    print vpc.state
    print vpc.region
    print vpc.tags

chapmanb

Topic:	Lorena Pantano: BIG meeting -- small RNA analysis (http://openwetware.org/wiki/BioMicroCenter:BIG_meeting)
Video:	https://youtu.be/l5nI8GICCPg
Slides:	https://github.com/lpantano/mypubs/blob/master/talks/BIG_Dec_15_2016.pdf
Time:	Dec 15, 2016 11:00 AM (GMT-5:00) Eastern Time (US and Canada)
Location:	MIT Koch Biology Building 68-156 (http://whereis.mit.edu/?go=68) or online
Online:	https://zoom.us/j/516853567
Telephone:	Dial: +1 646 558 8656 (US Toll) or +1 408 638 0968 (US Toll) International numbers available: https://zoom.us/zoomconference?m=yW4EVPMn0eK1ALOu2VlHPdzB4O3DS4Tp

chapmanb

--- command_line_HIV_pipeline.sh.orig	2016-12-02 15:10:02.856876302 -0500
+++ command_line_HIV_pipeline.sh	2016-12-03 21:15:21.750118228 -0500
@@ -8,18 +8,37 @@
 #$ -wd /path/to/working_directory
 #$ -t 1-400
 
+set -eu -o pipefail
+
+# Find original directory of bash script, resovling symlinks
+# http://stackoverflow.com/questions/59895/can-a-bash-script-tell-what-directory-its-stored-in/246128#246128

chapmanb

Test [6/87] kunkel 2016-11-23 22:09:06,431 MainThread ERROR root: Unsupported requirement InitialWorkDirRequirement
Test [39/87] Exception in thread Thread-8:
Traceback (most recent call last):
  File "/home/chapmanb/bio/toil/venv/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/chapmanb/bio/toil/venv/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/chapmanb/bio/toil/venv/lib/python2.7/site-packages/toil-3.5.0a1-py2.7.egg/toil/fileStore.py", line 1328, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

chapmanb

Blue Collar Bioinformatics (bcbio; http://bcb.io/) is community developed methods for variant calling, RNA-seq, small RNA and other biological analyses. We work together to build openly available, practical, validated tools for working with high throughput sequencing data.

At the MIT BIG meeting I'll discuss ongoing work to replace the infrastructure components of bcbio with the Common Workflow Language (CWL http://www.commonwl.org/). This allows integration of bcbio developed pipelines with multiple open source and commercial products. The goal is to improve how projects work together, giving bcbio users multiple infrastructure and hardware

chapmanb

{
    "region": "chrM:0-1000", 
    "vrn_file_region": {
        "class": "File", 
        "path": "/var/spool/cwl/gatk-haplotype/chrM/Test2-chrM_0_1000.vcf.gz", 
        "secondaryFiles": [
            {
                "class": "File", 
                "path": "/var/spool/cwl/gatk-haplotype/chrM/Test2-chrM_0_1000.vcf.gz.tbi"
            }

chapmanb

"""Trim start from sequences.

Usages:
    bcbio_python subset_fastq.py in.fastq num_bases > out.fastq
"""

import sys

from Bio import SeqIO