Created
November 13, 2012 07:24
-
-
Save MichaelXavier/4064471 to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| require 'celluloid' | |
| require './lib/discog_downloader/product' | |
| module DiscogDownloader | |
| class Converter | |
| include Celluloid | |
| def initialize(queue) | |
| @queue = queue | |
| end | |
| def convert(node) | |
| @queue << Product.new(:id => node['id']) | |
| end | |
| end | |
| end |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| require 'nokogiri' | |
| require 'celluloid' | |
| module DiscogDownloader | |
| class Parser | |
| include Celluloid | |
| attr_reader :reader, :next_pool | |
| def initialize(io, next_pool) | |
| @reader = Nokogiri::XML::Reader.from_io(io) | |
| @next_pool = next_pool | |
| end | |
| def parse! | |
| reader.each do |node| | |
| if node.name == "release" && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT | |
| next_pool.async.parse(node.outer_xml) | |
| end | |
| end | |
| # done. wat do? | |
| end | |
| end | |
| end |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| require 'nokogiri' | |
| require 'celluloid' | |
| module DiscogDownloader | |
| class Parser | |
| include Celluloid | |
| attr_reader :reader, :next_pool | |
| def initialize(io, next_pool) | |
| @reader = Nokogiri::XML::Reader.from_io(io) | |
| @next_pool = next_pool | |
| end | |
| def parse! | |
| reader.each do |node| | |
| if node.name == "release" && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT | |
| next_pool.async.parse(node.outer_xml) | |
| end | |
| end | |
| # done. wat do? | |
| end | |
| end | |
| end |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| require './lib/discog_downloader/node_parser' | |
| require './lib/discog_downloader/parser' | |
| require './lib/discog_downloader/converter' | |
| require 'thread' | |
| require 'benchmark' | |
| queue = Queue.new | |
| converter_pool = DiscogDownloader::Converter.pool(:args => [queue]) | |
| node_parser_pool = DiscogDownloader::NodeParser.pool(:args => [converter_pool]) | |
| parser = DiscogDownloader::Parser.new(ARGF, node_parser_pool) | |
| parser.async.parse! | |
| # wait for everyone to finish? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
it seems like your
NodeParserandParserare the same -- was that a copy/paste error? if this is all "fire and forget", the end of the line Celluloid actor (it looks likeConverterin this case) must complete the final task. is that to save them in a database after the xml is converted to raw attributes?