By sampling keys from your redis databases, this script tries to identify what types of keys are occupying the most memory.

simple_redis_profile.rb

#!/usr/bin/env ruby

# Evaluates a sample of keys/values from each redis database, computing statistics for each key pattern:
#   keys: number of keys matching the given pattern
#   size: approximation of the associated memory occupied (based on size/length of value)
#   percent: the proportion of this 'size' relative to the sample's total
#
# Copyright Weplay, Inc. 2010. Available for use under the MIT license.


require 'rubygems'
require 'redis'
require 'yaml'

SAMPLE_SIZE = 10_000  # number of keys to sample from each db before computing stats


# Naive approximation of memory footprint: size/length of value.
def redis_size(db, k)
  t = db.type(k)
  case t
    when 'string' then db.get(k).length
    when 'list'   then db.lrange(k, 0, -1).size
    when 'zset'   then db.zrange(k, 0, -1).size
    when 'set'    then db.smembers(k).size
    else raise("Redis type '#{t}' not yet supported.")  # TODO accommodate more types
  end
end

def array_sum(array)
  array.inject(0){ |sum, e| sum + e }
end

def redis_db_profile(db_name, sample_size = SAMPLE_SIZE)
  db = Redis.new(:db => db_name)
  keys = []
  sample_size.times { |i| keys << db.randomkey }
  key_patterns = keys.group_by{ |key| key.gsub(/\d+/, '#') }
  data = key_patterns.map{ |pattern, keys|
    [pattern, {'keys' => keys.size, 'size' => array_sum(keys.map{ |k| redis_size(db, k) })}]
  }.sort_by{ |a| a.last['size'] }.reverse
  size_sum = data.inject(0){|sum, d| sum += d.last['size'] }
  data.each { |d| d.last['percent'] = '%.2f%' % (d.last['size'].to_f*100/size_sum) }
end

db_names = `redis-cli info | grep ^db[0-9]`.split.map{ |line| line.scan(/^db\d+/).first }
db_names.each do |name|
  puts "\nProfiling \"#{name}\"...\n#{'-'*20}"
  y redis_db_profile(name)
end

puts "\nOverall statistics:\n#{'-'*20}"
puts `redis-cli info | grep memory`

Very nice! We have a bit more involved version that scans all keys, but using random key is better for analysis of a running master. We have all keys declared with something like this:

 redis_key :launch_finished, "jobs:JOB_ID:reports:ID:launch_finished"

where JOB_ID and ID are substituted. The analysis then aggregates the usage by a group.

Here's what I've done for size estimates that can improve your version:

  def key_size( redis, key )
    case redis.type(key)
    when "none" then 0
    when "string" then redis.get(key).size
    when "list" then redis.lrange(key,0,-1).map{|m| 1+m.size}.sum || 1
    when "zset" then redis.zrange(key,0,-1).map{|m| 1+m.size}.sum || 1
    when "set" then redis.smembers(key).map{|m| 1+m.size}.sum || 1
    when "hash" then redis.hgetall(key).to_a.flatten.map{|m| 1+m.size}.sum || 1
    end
  end