-
-
Save laserson/769114 to your computer and use it in GitHub Desktop.
| #! /usr/bin/env python | |
| import os | |
| import datetime | |
| import optparse | |
| import lsf | |
| option_parser = optparse.OptionParser() | |
| option_parser.add_option('-m','--make_jobs',action='store_true') | |
| option_parser.add_option('-s','--submit_jobs',action='store_true') | |
| option_parser.add_option('-q','--queue',default='normal_serial') | |
| option_parser.add_option('-l','--log_dir') | |
| (options,args) = option_parser.parse_args() | |
| # check that we get the qiime-required arguments | |
| if len(args) == 2: | |
| jobs_list_file = args[0] | |
| job_id = args[1] | |
| elif len(args) == 0: | |
| raise ValueError, "Didn't get the right command line arguments" | |
| # make a directory for holding LSF log files | |
| if options.log_dir == None: | |
| log_dir = os.path.join(os.environ['HOME'],'qiime_parallel_logs') | |
| else: | |
| log_dir = options.log_dir | |
| if not os.path.exists(log_dir): | |
| os.mkdir(log_dir,0755) | |
| # submit the jobs | |
| jobs_handle = open(jobs_list_file,'r') | |
| job_ids = [] | |
| logs = [] | |
| for (i,line) in enumerate(jobs_handle): | |
| datetimestamp = datetime.datetime.now().strftime('%Y%m%d_%H%M%S') | |
| log = os.path.join( log_dir, 'job_%i_%s.log' % (i,datetimestamp) ) | |
| job_id = lsf.submit_to_LSF(options.queue,log,line.strip()) | |
| job_ids.append(job_id) | |
| logs.append(log) | |
| jobs_handle.close() |
This script allows you to utilize the parallelization script in Qiime on a compute cluster that uses the LSF scheduling system.
When submitting jobs, LSF will email you upon completion by default. However, if Qiime is submitting many jobs, this can be quite cumbersome. This script will by default write the job reports out to $HOME/qiime_parallel_jobs. Currently, I don't think there is a way to specify this, though this is probably possible by using one of the special options to the parallel script. This is not implemented here.
Hi laserson,
I would like to use the script to submit parallel jobs to Platform LSF. I have copied them into my cluster and would like to spawn Qiime parallel jobs to Platform LSF. What else needs to be done to get this working? e.g. what changes do I need to ~/.qiime_config? Do I need to do anything else when submitting through bsub?
Thanks in advance.
The lsf.py module can be found here:
https://github.com/laserson/pytools/blob/master/lsf.py