explain.md

so, for the below script, we store the values in a text file that we read into mysql.

we then have 1000 unique URLs per row in a table, separated by a colon. We read and split one row at a time into a redis set, then pop values off the set whenever we need a new url. When the set is empty, we read the next row from mysql into the set.

The only thing to look out for is your collation in the db. A lookup is utf8_ci (case-insnsitive) for queries unless you set your collation to utf8_bin

collation-server = utf8_bin
init-connect='SET NAMES utf8'
init_connect='SET collation_connection = utf8_bin'
skip-character-set-client-handshake
character-set-server = utf8

perms.rb

##This script creates hashes
# it must be run from `rails console` because of
# the Time.now.to_s(:db), which is a rails thing.
#
# After this script is run, make the following sql call:
# load data infile '/tmp/perm_block.out' into table codepen.hash_seeds FIELDS TERMINATED BY ',' (hash_value,created_at,updated_at) SET id = NULL;
#
# The output of this call produces 60,233,040 unique hashes.  At 100k growth per month, this should last us 55 years.

puts 'gather and shuffle.'
perms = (('a'..'z').to_a + ('A'..'L').to_a).permutation(5).to_a
perms.shuffle!
puts 'shuffled'

open('/tmp/perm_block.out', 'w') do |f| 
  x = 0 
  s = []
  count = 1 
  perms.each do |p| 
    s << p.to_a.join
    x = x + 1 
    if x == 1000
      puts "writing #{count} of #{(perms.count / 1000).to_i}"
      f.puts [s.join(':'), Time.now.to_s(:db), Time.now.to_s(:db)].join(',')
      x = 0 
      count = count + 1 
      s.clear
    end 
  end 
end