Node JS Benchmarks Amazon AWS vs Joyent

gistfile1.md

The script:

var http = require('http')
var fs = require('fs')


http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'})
  res.end("hello world")
}).listen(3000);

This is just run with node server.js

The local ab

ab -n 100000 -c100 http://localhost:3000/

SmartOS High CPU 1.75 / SW-1

Requests per second: 1339.35 [#/sec] (mean)

Joyent High CPU 1.75 / East-1

Requests per second: 1019.17 [#/sec] (mean)

Joyent High CPU 1.75 Ubuntu 12.04

Requests per second: 2899.45 [#/sec] (mean)

Amazon High CPU 1.75 Ubuntu 12.04

Requests per second: 4085.26 [#/sec] (mean)

Python Loop

The code

for x in xrange(100000000):
    x + 1

The Test

time python test.py

SmartOS High CPU 1.75 / SW-1

real    0m13.531s
user    0m13.487s
sys     0m0.060s

Joyent High CPU 1.75 / East-1

real    0m10.337s
user    0m10.302s
sys     0m0.054s

Joyent High CPU 1.75 Ubuntu 12.04

real    0m9.177s
user    0m9.120s
sys     0m0.040s

Amazon High CPU 1.75

real    0m8.909s
user    0m8.897s                                                                                                          
sys     0m0.012s 

Notes

The first iterations of this had node clustered with 2 forks, incrementing a remote redis counter, and ab being run from a remote machine. While the numbers were higher ( about 50% ) they were proportionally the same. This version just removes all variables.

I understand ab isn't exact but it's margin of error is tight enough to make it suitable for a ballpark. I've worked with Joyent and Amazon in the past. Infact I pulled 100 machines off AWS and put them on Joyent precisely because my own benchmarks - much more thorough than this - showed Joyent being 5 - 10 times faster.

When I ran the above I expected Joyent to completely demolish AWS.

I'm happy to be told I've done something terribly wrong. In fact I'm hoping for it.

I'd be interested to see how wrk performs here too.

Also, just for anyone else who's tempted to copy/paste the Content-Length snippet - it should technically be 'Content-Length': Buffer.byteLength(body) - have been bitten by this before!

@mhart. Nice catch. Copy Paste is exactly what I did.

wrk looks interesting. Though it looks like it runs under the same paradigm as ab and zeusbench. One process on one machine opening n connections.

What we found was that the issue wasn't the benchmarking tool. ab and zb are booth good tools for what they do. The issue is in the the SmartOS TCP stack, it doesn't like getting a ton of requests from a single ip address. In order to get a reliable benchmark we had to reduce the concurrency and distribute the benchmark clients across multiple machines. 3 machines running at a concurrency of 33. jmeter, funkload, and tsung should all be able to perform this reasonably well. One could in theory fire ab, zb, wrk on muliple machines simultaneously and aggregate the results but the other tools I list already have that built in.