Skip to content

Instantly share code, notes, and snippets.

@nateware
Created October 25, 2012 22:47
Show Gist options
  • Save nateware/3955947 to your computer and use it in GitHub Desktop.
Save nateware/3955947 to your computer and use it in GitHub Desktop.
S3 file hashing scheme for linear distribution
#!/bin/sh
# Get crazy throughput if you do this (>5000 puts/sec)
#
# What some customers have done is something like this: say this is the filename:
#
# mybucket/17476/26466/16.png
#
# Then to get the hash do something that seems silly but works great (pretty
# much any two characters from the hash will work):
#
# echo '/17476/26466/' | openssl dgst -md5 -hex | cut -c22-23
#
# So this key ends up being:
#
# mybucket/6e/17476/26466/16.png
#
echo $filename | cut -d/ -f1 | openssl dgst -md5 -hex | cut -c22-23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment