Processing large files with hamster.rdoc

And finally, you also get IO#to_list allowing you to lazily processes huge files. For example, imagine the following code to process a 10MB file:

File.open("my_10_mb_file.txt") do |io|
  lines = []
  io.each_line do |line|
    break if lines.size == 10
    lines << line.chomp.downcase.reverse
  end
end

How many times/how long did you read the code before it became apparent what the code actually did? Now compare that to the following:

File.open("my_10_mb_file.txt") do |io|
  io.map(&:chomp).map(&:downcase).map(&:reverse)[0, 10]
end

Unfortunately, though the second example reads nicely, it takes around 3 seconds to run (compared with 0.033 seconds for the first) even though we’re only interested in the first 10 lines! However, using a little #to_list magic, we can get the running time down to 0.033 seconds!

File.open("my_10_mb_file.txt") do |io|
  io.to_list.map(&:chomp).map(&:downcase).map(&:reverse).take(10)
end

How is this even possible? It’s possible because IO#to_list creates a lazy list whereby each line is only ever read and processed as needed, in effect converting it to the first example without all the syntactic, imperative, noise.