Hello world of Sinatra-like libraries on Elixir, Ruby and Node.js

0_README.markdown

Lies, Damned Lies and Benchmarks

This is just an exercise to measure the performance between Sinatra-like libraries in:

• Elixir v0.4.dev (running on top of the Erlang VM R14B03) with Dynamo;

• Ruby 1.9.2 with [Sinatra Synchrony] v0.0.3 (https://github.com/kyledrake/sinatra-synchrony) + Thin;

• node.js 0.4.7 with [express] v2.3.10(https://github.com/visionmedia/express)

The basic example uses only a single route at /hello_world and return Hello world. These benchmarks are extremely simple and should not be used as a decision factor for any of the tools presented (unless you are building hello world apps ;). I am just using them as performance guideline when working on Dynamo.

What happens if we add more routes?

This benchmark also measures what happens if you add 10, 100 and 1000 /hello_world/#{i} routes and try to match the last /hello_world/1000 route.

What happens if I am serving a file?

We also check the behavior of the app when it has to serve a file with 100kb of size.

What happens if I implement a Fibonacci server?

Just trolling bro.

Results

Results were obtained by running the examples below in a Macbook Pro 2011 2.3GHz dual-core Intel Core i5. No customization was done to the underlying VM/Engine/Webserver besides passing the "production" flag to all libraries, i.e. I am measuring the out-of-the-box experience. This means Elixir is running on both cores, while the other solutions aren't. This is one of the several benefits of using the Erlang VM.

req/s average for 8000 requests, concurrency 40 (ab -n 8000 -c 40) (more is better):

   Routes     | Dynamo/Elixir | SinatraS/Ruby 1.9.2 | Express/node.js 0.4.7
      1       |    9135.83    |      3833.54        |      4730.74
   10 + 1     |    9105.35    |      3561.69        |      4635.8
  100 + 1     |    9103.91    |      2472.51        |      4171.94
 1000 + 1     |    9043.72    |       598.47        |      1904.71
 100kb file   |    6087.99    |      2088.66        |      1734.17

Notes

Thanks to the Erlang VM, Dynamo performs very well in the basic hello world case. Indeed, Dynamo is in very early stage and the throughput will reduce by a couple dozens req/s as we add more basic features. Regardless, very good results.

It is also important to notice that increasing the number of routes increases the CPU usage which will block on single process, non threaded solutions like Node.js, affecting throughput. Also the performance of the algorithm used in the routes matching matters in the results above.

In the future, I plan to draw graphs on how those examples behave when increasing the concurrency. In early experiments, both Sinatra and Express throughput starts to decrease early on when increasing the concurrency. I also want to run benchmarks against those servers deployed on Heroku, but not sure if i need their permission first or anything. :)

1_elixir_dynamo.exs

defmodule MyApp do
  use Dynamo::Routes

  get "/hello_world" do
    "Hello World"
  end
end

Dynamo::Cowboy.run MyApp

2_ruby_sinatra_synchrony.rb

require 'sinatra/base'
require 'sinatra/synchrony'

class HelloWorld < Sinatra::Base
  register Sinatra::Synchrony

  get '/hello_world' do
    'Hello World'
  end
end

3_node_express.js

var app = require('express').createServer();

app.get('/hello_world', function(req, res){
  res.send('hello world');
});

app.listen(3000);

Promising results. Are You planning to prepare whole non-blocking stdlib for Elixir?
What is Dynamo? The README isn't descriptive.
Numbers are OK for regular apps but if You can lift them even more it could be agile competitor for libevent or for example LMAX Disruptor on APIs and services field.

Everything in Erlang and therefore in Elixir is non blocking. Everything works transparently, you don't need to write callback/asynchronous code. Another benefit of Erlang is that it can hold an extremely high amount of connections in the same node. For a web sockets/polling application, it is easy to find reports of 500.000 connections on a single node. I am on mobile now, but I want to write more detailed reports later.

Is there a typo on the benchmarks for Sinatra with a 1000 + 1? It's displaying 598.47, is that correct? Besides that, perhaps you should put a "higher numbers are better" message for the casual readers :)

Not a typo. :) And I added a "more is better" note. Thanks!

Also, in many cases ab is broken on OS X. I found worth trying to run benchmarks on linux (I'm using OSX Lion by default) which leads to different results. Probably You have good version of ab but to be real sure about your numbers, what was complete / failed requests ratio?

@burnhype I don't have this information anymore. I hope to re-run those benches in the future.

@josevalim Hey Jose, any chance of a new benchmark for the latest versions? :)

@josevalim I think NodeJS would perform better than Elixir after clustering.

@abhijitiitr - Defeats the purpose of out of box if he has to go add a bunch of extensions, no?

@wess it does...but does it really matter since clustering is commonly used in all nodejs apps and has become part of the standard code in nodejs

@abhijitiitr if 1 instance of elixir can beat 1 instance of node.js in any scenario, why would cluster node.js perform better than elixir?

Um, I actually do have a Node+Express Fibonacci service...
https://github.com/jocull/node-fibonacci-service