Benchmarks for stripe_event

benchmarks.md

Benchmarks

This document is a basic comparison of the performance of the stripe_event gem given different Rails Controller setups. StripeEvent::WebhookController only needs the head method provided by the ActionController::Head module. This means the controller can inherit from ActionController::Metal rather than ActionController::Base if ActionController::Head is included. I'd like to have some data to support this decision.

Performance will be measured with the Apache HTTP server benchmarking tool using the unix utility ab. The ab that ships with OS X Lion seems to be broken, so I installed the version with Homebrew: brew install ab.

The benchmarks will be run on a MacBook Air with the following specs:

• OS X Lion 10.7.5 (11G63b)
• 1.7 GHz Intel Core i5
• 4 GB 1333 MHz DDR3

Environment:

$ ruby -v
ruby 1.9.3p385 (2013-02-06 revision 39114) [x86_64-darwin11.4.2]
$ rails -v
Rails 3.2.12

The command used to take the measurements is:

$ ab -n 1000 -T "application/json" \
>            -p benchmark.json \
>            -r http://localhost:3000/stripe_event

The benchmark.json file contains the minimum parameter set required to successfully execute the request:

{ "type": "customer.created" }

In the dummy application that the gem is tested against, StripeEvent::Engine is mounted at the path /stripe_event.

# spec/dummy/config/routes.rb
Rails.application.routes.draw do
  mount StripeEvent::Engine => "/stripe_event"
end

The default behavior of the gem is to retrieve the Stripe::Event object via Stripe's api. This is not acceptable for benchmarking so that particular behavior will need to be changed:

# lib/stripe_event.rb
# before
self.event_retriever = lambda { |params| Stripe::Event.retrieve(params[:id]) }
# after
self.event_retriever = lambda { |params| params }

Here is the testing process:

$ cd spec/dummy
$ rails s -e production -d
$ ab -n 1000 -T "application/json" -p benchmark.json -r http://localhost:3000/stripe_event
# Results taken from the second run
$ ab -n 1000 -T "application/json" -p benchmark.json -r http://localhost:3000/stripe_event

Current Setup

module StripeEvent
  class WebhookController < ActionController::Base
    def event
      StripeEvent.instrument(params)
      head :ok
    end
  end
end

Results:

Server Software:        WEBrick/1.3.1
Server Hostname:        localhost
Server Port:            3000

Document Path:          /stripe_event
Document Length:        1 bytes

Concurrency Level:      1
Time taken for tests:   4.849 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      413000 bytes
Total body sent:        175000
HTML transferred:       1000 bytes
Requests per second:    206.22 [#/sec] (mean)
Time per request:       4.849 [ms] (mean)
Time per request:       4.849 [ms] (mean, across all concurrent requests)
Transfer rate:          83.17 [Kbytes/sec] received
                        35.24 kb/s sent
                        118.41 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     4    5   1.7      4      18
Waiting:        4    4   1.7      4      17
Total:          4    5   1.7      4      18

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      5
  80%      5
  90%      5
  95%      6
  98%     15
  99%     15
 100%     18 (longest request)

New Setup

module StripeEvent
  class WebhookController < ActionController::Metal
    include ActionController::Head

    def event
      StripeEvent.instrument(params)
      head :ok
    end
  end
end

Results:

Server Software:        WEBrick/1.3.1
Server Hostname:        localhost
Server Port:            3000

Document Path:          /stripe_event
Document Length:        1 bytes

Concurrency Level:      1
Time taken for tests:   4.311 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      389000 bytes
Total body sent:        175000
HTML transferred:       1000 bytes
Requests per second:    231.97 [#/sec] (mean)
Time per request:       4.311 [ms] (mean)
Time per request:       4.311 [ms] (mean, across all concurrent requests)
Transfer rate:          88.12 [Kbytes/sec] received
                        39.64 kb/s sent
                        127.76 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     3    4   2.1      4      22
Waiting:        3    4   2.0      4      22
Total:          4    4   2.1      4      22

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      4
  80%      4
  90%      4
  95%      5
  98%     14
  99%     16
 100%     22 (longest request)