Skip to content

Instantly share code, notes, and snippets.

@johanb
Created April 5, 2012 13:36
Show Gist options
  • Select an option

  • Save johanb/2311073 to your computer and use it in GitHub Desktop.

Select an option

Save johanb/2311073 to your computer and use it in GitHub Desktop.
Just zipping up some photos.. this leaks memory. How can I optimize this routine ?
photos.each do |photo|
path = photo.photo.path
file_path = zip_path + "/" + path
FileUtils.mkdir_p(File.dirname(file_path))
File.open(file_path, 'wb') do |file|
file.write photo.photo.file.read
file.close # shouldn't make a difference
end
# FWIW, this doesn't make a difference
# file = File.open(file_path, 'wb')
# file.write photo.photo.file.read
# file.close
end
system("cd #{temp_dir} && zip #{zip_path} -r #{zip_name}")
# This seem sot be my solution of avoiding a huge amount of memory usage. (Thanks to [hanke](http://twitter.com/hanke) & [alloy](http://twitter.com/alloy) )
FileUtils.mkdir_p(zip_path)
photos.each do |photo|
url = photo.photo_url
path = photos.photo.path
system("cd #{zip_path} && curl -L #{url} -o #{path} --create-dirs")
end
system("cd #{temp_dir} && zip #{zip_path} -r #{zip_name}")
# This was my initial attempt which was altered transformed to what can be seen in Downloading in Ruby Zipping Shelled Out.rb
Zip::ZipOutputStream.open(zip_path) do |zos|
photos.each do |photo|
zos.put_next_entry(photo.photo.path)
zos.write photo.photo.file.read
end
end
@johanb

johanb commented Apr 10, 2012

Copy link
Copy Markdown
Author
  • According to the docs I should not have to call "file.close" inside the File.open block since it should close the file automatically.
  • Shelled Out.rb still increases memory usage until an unworkable high level (500 MB which is very close the the size of the images being loaded).
  • When inspecting the process while running with Activity Monitor no more then 1 file is open at a time. So my assumption is this is not the problem.
  • The memory increases before the zipping ever takes place so this isn't the problem either.
  • All associations of photo are eagerly loaded which should therefor not be the problem either (and the DB is no where near 500 MB)
  • The memory pain is obviously in the photo.photo.file.read call.
  • The photo.photo.file.read call comes from this file in CarrierWave Fog
  • Turns out the photo.photo.file.read call caches every single file read. Which prevents GC.

@floere

floere commented Apr 11, 2012

Copy link
Copy Markdown

Just a note to the last line here: It only prevents GC'ing of the downloaded file body because there's a reference chain to the file. A clear method would be nice, or similar.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment