var http = require('http')
var fs = require('fs')
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'})
res.end("hello world")
}).listen(3000);
This is just run with node server.js
ab -n 100000 -c100 http://localhost:3000/
Requests per second: 1339.35 [#/sec] (mean)
Requests per second: 1019.17 [#/sec] (mean)
Requests per second: 2899.45 [#/sec] (mean)
Requests per second: 4085.26 [#/sec] (mean)
for x in xrange(100000000):
x + 1
time python test.py
real 0m13.531s user 0m13.487s sys 0m0.060s
real 0m10.337s user 0m10.302s sys 0m0.054s
real 0m9.177s user 0m9.120s sys 0m0.040s
real 0m8.909s user 0m8.897s sys 0m0.012s
The first iterations of this had node clustered with 2 forks, incrementing a remote redis counter, and ab being run from a remote machine. While the numbers were higher ( about 50% ) they were proportionally the same. This version just removes all variables.
I understand ab isn't exact but it's margin of error is tight enough to make it suitable for a ballpark. I've worked with Joyent and Amazon in the past. Infact I pulled 100 machines off AWS and put them on Joyent precisely because my own benchmarks - much more thorough than this - showed Joyent being 5 - 10 times faster.
When I ran the above I expected Joyent to completely demolish AWS.
I'm happy to be told I've done something terribly wrong. In fact I'm hoping for it.
The benchmark isn't testing what it may seem to test. It's launching a flood of TCP connections from a single client IP address, which is causing a problem.
On SmartOS, these new connections are clashing with old ones still in TIME_WAIT, causing client retransmits and delays. The problem is that TCP ports are 16-bit, and the four-tuple (or three-tuple) of client-IP:client-port:server-IP:server-port is used as a unique TCP connection identifier. Since this benchmark only has one client IP, one server IP, and one server port, the only variable to uniquely identify connections is the 16-bit client ephemeral port (which by default is restricted to 32k-65k, so only 15-bits). So after (only) tens of thousands of connections, the chances of colliding with an old session in TIME_WAIT become great, which slows down ab. Linux has a different mechanism for recycling TIME_WAIT sessions, which is better at avoiding this.
While the test is running on SmartOS, the node server is largely idle:
In that interval, node is spending 83% of its time sleeping, as the ab clients becomes blocked on TCP retransmits.
The best way to benchmark node.js is using multiple clients, which avoids the TIME_WAIT clashes, and is what usually happens in the real world.
I'd like to show an easy workaround for anyone trying this form of single client benchmarking: have the node server listen on several ports, and have ab test them all simultaneously, with the aim of driving the node CPU time to 100%. However, I don't see a way to have ab test more than one URL at a time. :(
Here's a different tactic: adjust the server to support keep-alive:
Then use ab -k
The idea of using keep-alive is to take the single-client-connection-flood-problem off the table.
Again, the use of multiple clients should avoid the clash, as usually happens in production.