The point I was trying to make is that the differences look large because it's a hello world. I understand that you're benchmarking the overhead of the servers, but that doesn't mean anything until you compare the overhead to how much time the app itself needs.
The benefit of benchmarking the network is:
- The overhead of the benchmarking program itself is not affecting the results.
- It gives you a more realistic picture, because in production environments the network delays are significant. In fact, they should be so significant that the differences between the app servers should be greatly reduced.
Therefore, using Vagrant and VMs on the local machine will not work. You really have to benchmark over a network, using another computer. In my test I grabbed another laptop. You might want to spin up an EC2 instance.
A while ago we benchmarked Unicorn vs Passenger 4 over the network. The performance is almost identical: https://code.google.com/p/phusion-passenger/issues/detail?id=956#c4
But I'm curious: what server configurations did you use? It matters a lot.
That is super interesting, but not completely surprising.
I know that once you throw in a network, database, etc. the basic routing and application server performance is almost never the bottleneck.
I'm not sure how easy it would be to test over a network simply because then you're limited by the bandwidth and connections between your local provider and the server on EC2. I don't know if running on EC2 you'd be testing your local bandwidth more than the app server performance, thus you are in effect not testing the app server differences at all, but rather testing network performance.
I don't know if anyone has good examples of how to test over the network without being bandwidth constrained. Have you seen any?