Batch download genbank files from IRES website

Unix batch download

1. Created a script for stripping out the urls for genbank files.

#IRES_PLASMID_SCRAPER.rb

pwd = "/Users/timrand/Dropbox/NAT1/"
file = pwd + "IRES_plasmids.txt"
website_url = "http://iresite.org/IRESite_web.php?page=gb_download&file="
out_file = pwd + "ires_plasmid_urls.txt"

f = File.read(file)
plasmid_files = f.scan(/\w+\.gb/)
File.open(out_file, 'w') do |o|
  plasmid_files.each{ |pl| o.puts(website_url + pl) }
end

     
% head ires_plasmid_urls.txt http://iresite.org/IRESite_web.php?page=gb_download&file=4.gb
http://iresite.org/IRESite_web.php?page=gb_download&file=7.gb
http://iresite.org/IRESite_web.php?page=gb_download&file=10.gb
http://iresite.org/IRESite_web.php?page=gb_download&file=13.gb
http://iresite.org/IRESite_web.php?page=gb_download&file=16.gb
http://iresite.org/IRESite_web.php?page=gb_download&file=19.gb
http://iresite.org/IRESite_web.php?page=gb_download&file=24.gb
http://iresite.org/IRESite_web.php?page=gb_download&file=27.gb
http://iresite.org/IRESite_web.php?page=gb_download&file=30.gb
http://iresite.org/IRESite_web.php?page=gb_download&file=33.gb

2. Unix allows you to set the basename of a file. It will be appended wherever a filename is used.

% basename="http://iresite.org/IRESite_web.php?page=gb_download&file="
% echo $basename
http://iresite.org/IRESite_web.php?page=gb_download&file=


3. open urls.txt, read line by line, curl for the webpage, and save to a file named after the plasmid number.

% cat urls.txt | while read a; do curl -v --proxy proxy.kuins.net:8080 "$a" > `echo $a | egrep -o '[0-9]+\.gb'`; done