Skip to content

Instantly share code, notes, and snippets.

@ajbonner
Last active December 12, 2015 08:39
Show Gist options
  • Save ajbonner/4745978 to your computer and use it in GitHub Desktop.
Save ajbonner/4745978 to your computer and use it in GitHub Desktop.
Multithreaded Ruby Rogue Podcast Slurper
#!/usr/bin/env ruby
require 'nokogiri'
require 'open-uri'
MAX_THREADS = 5
thread_pool = []
index = Nokogiri::HTML(open('http://rubyrogues.com/episode-guide/'));
index.css('a[rel=bookmark]').each do |link|
while thread_pool.size >= MAX_THREADS
Thread.list.each do |t|
thread_pool.delete!(t) unless t.alive?
sleep(1)
end
end
t = Thread.new do
podcast = Nokogiri::HTML(open(link['href']));
title = podcast.css('.entry-title').text
clean_title = title.gsub(/[^a-zA-Z0-9]/, '-') + '.mp3'
clean_title.gsub!(/-{2,}/, '-')
download_link = podcast.css('a[title=Download]')
%x[wget -q -O '#{clean_title}' '#{download_link[0]['href']}']
end
thread_pool.push(t)
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment