Run ncdu over a listing of an S3 bucket

README.md

s3-ls-to-ncdu

Run ncdu over a listing of an S3 bucket.

How to

1. With a working AWS CLI setup, run aws s3 ls s3://example-bucket --recursive | tee bucket-listing.txt to dump the bucket's contents.
2. Run this script and save its output to another file ruby s3-ls-to-ncdu.rb > bucket-listing.ncdu.json.
3. Run ncdu -f bucket-listing.ncdu.json to see what's going on in your bucket.

Bugs & improvements

I've just made this to work for my one interesting bucket. Feel free to add more features or report bugs you may have.

s3-ls-to-ncdu.rb

#!/usr/bin/env ruby
require 'json'

# The listing is the output of `aws s3 ls s3://example-bucket --recursive | tee bucket-listing.txt`
listing = File.read('./bucket-listing.txt').freeze

files = []
listing.each_line do |line|
  file_size = line[19..29].strip.to_i
  path = line[30..].strip

  files << { name: path, size: file_size } if file_size.positive?
end

root = {}

files.each do |file|
  current_node = root
  file[:name].split('/').each do |part|
    current_node[part] ||= {}
    current_node = current_node[part]
  end
  current_node[:size] = file[:size]
end

def ncdu_format(path, content)
  if content.key? :size
    { name: path, dsize: content[:size] }
  else
    subcont = [
      { name: path }
    ]
    content.each { |key, value| subcont << ncdu_format(key, value) }
    subcont
  end
end

contents = [
  { name: 'example-bucket' } # bucket root
]

root.each_pair do |key, content|
  contents << ncdu_format(key, content)
end

output = [
  1, # major version
  0, # minor version
  {}, # metadata, don't care
  contents
]

puts output.to_json