How Shopify Scales Rails John Duff

The Stack:

ruby1.9.3-p327
rails3.2
unicorn 4.5
percona Mysql5.5
memcache14.14
redis2.6

33 app servers, 1172 unicorn workers, 5 job servers, 370 job workers 211 controllers, 468 models..

Current Scale:

9.9 M orders last year 2008 sales per minutes Cyber Monday 50,000 RPM, 45ms response time

Looking back:

first line of code written in 2004 shopify released on July, 2005

Know where to optimize the system. one request, one process

increase workers, discrease reponse time

Know the system

avoid network calls during requests
speed up unavoidable network calls
the storefront and checkout
the Chive: handle spikes in the system

Measure ALL THE THINGS

New Relic
splunk
statsD
Cacti //mysql records historic stats
Conan //test production system

dashboard all over the office

Caching

cacheable https://github.com/Shopify/cacheable

serve gzip'd content, stored in memcache. increase the response
generational caching
no explicit expiry

caching 404s

Identity Cache: https://github.com/Shopify/identity_cache

cache full model objects in memcached
can include associated objects in cache
must opt in to the cache
explicit, but expiry

class Product < ActiveRecord::Base
  include IdentityCache

  has_many :images

  cache_has_many :images, :embed => true
end

# Fetch the product by its id, the primary index.
@product = Product.fetch(id)

# Fetch the images for the Product. Images are embedded so the product fetch would have already loaded them.
@images = @product.fetch_images

Get out of my process

Delayed Job

jobs stored in the db
workers run in their own process
workers poll for jobs periodically

moved to:

Resque: https://github.com/resque/resque

redis backed
no more db contention
faster (300 jobs/sec vs 120 jobs/sec)
extensible

ex:

sending emails
processing payments
geolocation
import/export

Class AddressGeolocationJob
max_retries 3

def sefl.perform(params)
  object = params[:model].constantize.find(params[:id])
  object.latitude, object.longtitude = Geocoder.geocode(object)
  object.save!
end
end

Resque.enque(AddressGeolocationJob, :id => 1, :model => 'Address')

Redis

Inventory resavation system
sessions
theme uploads
throttliing
sequenced column

Speed up MySQL

4x8 core processor
SSD
256 GB RAM
full working set in memory

query opt

pt-query-digest
avoid queries that generate temp tables
adding the right indexes
forcing/ignoring indexes

MySQL tuning

disable innodb_stats_on_metadata
increase table_open_cache
replace glibc memory allocator with tcmalloc
innodb_autoinc_lock_mode='interleaved'

after_commit //db transactions

after transaction has been commmited
webhooks
cache expiry
update associated objects

Services

split out standalone services as needed
independently scaled
segmented metrics
overall system is more complex ex) Imagery

@johnduff http://www.youtube.com/watch?v=j347oSSuNHA

naoyamakino/railsConfShopifyScaling.md