Skip to content

Instantly share code, notes, and snippets.

@MichaelXavier
Created November 13, 2012 07:24
Show Gist options
  • Select an option

  • Save MichaelXavier/4064471 to your computer and use it in GitHub Desktop.

Select an option

Save MichaelXavier/4064471 to your computer and use it in GitHub Desktop.
require 'celluloid'
require './lib/discog_downloader/product'
module DiscogDownloader
class Converter
include Celluloid
def initialize(queue)
@queue = queue
end
def convert(node)
@queue << Product.new(:id => node['id'])
end
end
end
require 'nokogiri'
require 'celluloid'
module DiscogDownloader
class Parser
include Celluloid
attr_reader :reader, :next_pool
def initialize(io, next_pool)
@reader = Nokogiri::XML::Reader.from_io(io)
@next_pool = next_pool
end
def parse!
reader.each do |node|
if node.name == "release" && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT
next_pool.async.parse(node.outer_xml)
end
end
# done. wat do?
end
end
end
require 'nokogiri'
require 'celluloid'
module DiscogDownloader
class Parser
include Celluloid
attr_reader :reader, :next_pool
def initialize(io, next_pool)
@reader = Nokogiri::XML::Reader.from_io(io)
@next_pool = next_pool
end
def parse!
reader.each do |node|
if node.name == "release" && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT
next_pool.async.parse(node.outer_xml)
end
end
# done. wat do?
end
end
end
require './lib/discog_downloader/node_parser'
require './lib/discog_downloader/parser'
require './lib/discog_downloader/converter'
require 'thread'
require 'benchmark'
queue = Queue.new
converter_pool = DiscogDownloader::Converter.pool(:args => [queue])
node_parser_pool = DiscogDownloader::NodeParser.pool(:args => [converter_pool])
parser = DiscogDownloader::Parser.new(ARGF, node_parser_pool)
parser.async.parse!
# wait for everyone to finish?
@jeremyvdw

Copy link
Copy Markdown

Looks like you paste parser.rb content for node_parser.rb file

@cookrn

cookrn commented Nov 16, 2012

Copy link
Copy Markdown

it seems like your NodeParser and Parser are the same -- was that a copy/paste error? if this is all "fire and forget", the end of the line Celluloid actor (it looks like Converter in this case) must complete the final task. is that to save them in a database after the xml is converted to raw attributes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment