Qiime parallel job submission script for LSF queuing system

qiime_cluster_jobs_LSF.py

#! /usr/bin/env python

import os
import datetime
import optparse

import lsf

option_parser = optparse.OptionParser()
option_parser.add_option('-m','--make_jobs',action='store_true')
option_parser.add_option('-s','--submit_jobs',action='store_true')
option_parser.add_option('-q','--queue',default='normal_serial')
option_parser.add_option('-l','--log_dir')
(options,args) = option_parser.parse_args()

# check that we get the qiime-required arguments
if len(args) == 2:
    jobs_list_file = args[0]
    job_id = args[1]
elif len(args) == 0:
    raise ValueError, "Didn't get the right command line arguments"

# make a directory for holding LSF log files
if options.log_dir == None:
    log_dir = os.path.join(os.environ['HOME'],'qiime_parallel_logs')
else:
    log_dir = options.log_dir

if not os.path.exists(log_dir):
    os.mkdir(log_dir,0755)

# submit the jobs
jobs_handle = open(jobs_list_file,'r')
job_ids = []
logs = []
for (i,line) in enumerate(jobs_handle):
    datetimestamp = datetime.datetime.now().strftime('%Y%m%d_%H%M%S')
    log = os.path.join( log_dir, 'job_%i_%s.log' % (i,datetimestamp) )
    job_id = lsf.submit_to_LSF(options.queue,log,line.strip())
    job_ids.append(job_id)
    logs.append(log)
jobs_handle.close()

This script allows you to utilize the parallelization script in Qiime on a compute cluster that uses the LSF scheduling system.

When submitting jobs, LSF will email you upon completion by default. However, if Qiime is submitting many jobs, this can be quite cumbersome. This script will by default write the job reports out to $HOME/qiime_parallel_jobs. Currently, I don't think there is a way to specify this, though this is probably possible by using one of the special options to the parallel script. This is not implemented here.

Hi laserson,

I would like to use the script to submit parallel jobs to Platform LSF. I have copied them into my cluster and would like to spawn Qiime parallel jobs to Platform LSF. What else needs to be done to get this working? e.g. what changes do I need to ~/.qiime_config? Do I need to do anything else when submitting through bsub?

Thanks in advance.

It's been a while since I messed with it, but the documentation is relatively clear here: http://qiime.sourceforge.net/tutorials/parallel_qiime.html Also note that the LSF command that my function calls is `rbsub`. I don't know if every LSF system has that, so you can just replace it with `bsub` and it should work the same. Good luck! Uri ................................................................................... Uri Laserson Graduate Student, Biomedical Engineering Harvard-MIT Division of Health Sciences and Technology M +1 917 742 8019 laserson@mit.edu

…

On Tue, Nov 8, 2011 at 05:30, miahw < ***@***.***>wrote: Hi laserson, I would like to use the script to submit parallel jobs to Platform LSF. I have copied them into my cluster and would like to spawn Qiime parallel jobs to Platform LSF. What else needs to be done to get this working? e.g. what changes do I need to ~/.qiime_config? Do I need to do anything else when submitting through bsub? Thanks in advance. --- Reply to this email directly or view it on GitHub: https://gist.github.com/769114