Skip to content

Instantly share code, notes, and snippets.

@courtneycouch

courtneycouch/Raw results

Created May 11, 2012 03:53
Show Gist options
  • Save courtneycouch/2657432 to your computer and use it in GitHub Desktop.
Save courtneycouch/2657432 to your computer and use it in GitHub Desktop.
Download ZIP
AB test results - nodejs vs vert.x
Raw
Raw results
Single core - ab test single concurrency 100k requests
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /
Document Length: 89 bytes
Concurrency Level: 1
Time taken for tests: 26.165080 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 17200000 bytes
HTML transferred: 8900000 bytes
Requests per second: 3821.89 [#/sec] (mean)
Time per request: 0.262 [ms] (mean)
Time per request: 0.262 [ms] (mean, across all concurrent requests)
Transfer rate: 641.92 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 0 0 0.2 0 9
Waiting: 0 0 0.1 0 9
Total: 0 0 0.2 0 9
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 0
99% 0
100% 9 (longest request)
Single core - ab test 10x concurrency 100k requests
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /
Document Length: 89 bytes
Concurrency Level: 10
Time taken for tests: 18.556226 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 17200000 bytes
HTML transferred: 8900000 bytes
Requests per second: 5389.03 [#/sec] (mean)
Time per request: 1.856 [ms] (mean)
Time per request: 0.186 [ms] (mean, across all concurrent requests)
Transfer rate: 905.14 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 1 1 0.5 1 11
Waiting: 0 1 0.5 1 11
Total: 1 1 0.5 1 11
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 1
95% 1
98% 1
99% 2
100% 11 (longest request)
Single core - ab test 100x concurrency 100k requests
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /
Document Length: 89 bytes
Concurrency Level: 100
Time taken for tests: 18.239243 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 17200000 bytes
HTML transferred: 8900000 bytes
Requests per second: 5482.68 [#/sec] (mean)
Time per request: 18.239 [ms] (mean)
Time per request: 0.182 [ms] (mean, across all concurrent requests)
Transfer rate: 920.87 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 4
Processing: 0 17 6.3 17 35
Waiting: 0 17 6.3 17 35
Total: 2 17 6.3 17 35
Percentage of the requests served within a certain time (ms)
50% 17
66% 23
75% 23
80% 23
90% 23
95% 24
98% 29
99% 31
100% 35 (longest request)
Single core - ab test 1000x concurrency 100k requests
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /
Document Length: 89 bytes
Concurrency Level: 1000
Time taken for tests: 17.803068 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 17200000 bytes
HTML transferred: 8900000 bytes
Requests per second: 5617.01 [#/sec] (mean)
Time per request: 178.031 [ms] (mean)
Time per request: 0.178 [ms] (mean, across all concurrent requests)
Transfer rate: 943.43 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 79 598.3 0 9001
Processing: 3 43 24.8 44 2074
Waiting: 3 43 24.7 44 2074
Total: 3 123 602.4 44 9066
Percentage of the requests served within a certain time (ms)
50% 44
66% 44
75% 45
80% 46
90% 52
95% 55
98% 3030
99% 3047
100% 9066 (longest request)
8 core - ab test 1x concurrency 100k requests
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /
Document Length: 89 bytes
Concurrency Level: 1
Time taken for tests: 27.673151 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 17200000 bytes
HTML transferred: 8900000 bytes
Requests per second: 3613.61 [#/sec] (mean)
Time per request: 0.277 [ms] (mean)
Time per request: 0.277 [ms] (mean, across all concurrent requests)
Transfer rate: 606.94 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 4
Processing: 0 0 0.1 0 9
Waiting: 0 0 0.1 0 9
Total: 0 0 0.1 0 9
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 0
99% 0
100% 9 (longest request)
8 core - ab test 10x concurrency 100k requests
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /
Document Length: 89 bytes
Concurrency Level: 10
Time taken for tests: 8.602445 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 17200000 bytes
HTML transferred: 8900000 bytes
Requests per second: 11624.60 [#/sec] (mean)
Time per request: 0.860 [ms] (mean)
Time per request: 0.086 [ms] (mean, across all concurrent requests)
Transfer rate: 1952.47 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 4
Processing: 0 0 0.5 0 12
Waiting: 0 0 0.2 0 11
Total: 0 0 0.5 0 12
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 1
95% 1
98% 1
99% 1
100% 12 (longest request)
8 core - ab test 100x concurrency 100k requests
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /
Document Length: 89 bytes
Concurrency Level: 100
Time taken for tests: 9.98349 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 17200172 bytes
HTML transferred: 8900089 bytes
Requests per second: 10991.01 [#/sec] (mean)
Time per request: 9.098 [ms] (mean)
Time per request: 0.091 [ms] (mean, across all concurrent requests)
Transfer rate: 1846.16 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.3 1 9
Processing: 1 7 1.5 7 21
Waiting: 0 2 1.6 3 17
Total: 1 8 2.0 8 23
Percentage of the requests served within a certain time (ms)
50% 8
66% 9
75% 10
80% 10
90% 11
95% 12
98% 13
99% 14
100% 23 (longest request)
8 core - ab test 1000x concurrency 100k requests
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /
Document Length: 89 bytes
Concurrency Level: 1000
Time taken for tests: 9.831740 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 17201032 bytes
HTML transferred: 8900534 bytes
Requests per second: 10171.14 [#/sec] (mean)
Time per request: 98.317 [ms] (mean)
Time per request: 0.098 [ms] (mean, across all concurrent requests)
Transfer rate: 1708.45 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 15 9.9 15 83
Processing: 16 81 11.0 82 169
Waiting: 0 36 10.3 36 153
Total: 22 97 17.3 96 251
Percentage of the requests served within a certain time (ms)
50% 96
66% 103
75% 108
80% 111
90% 118
95% 126
98% 135
99% 141
100% 251 (longest request)
ab -c 1 -n 100000 http://127.0.0.1:8080/
Vert.X
8 core - ab test 1x concurrency 100k requests
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /
Document Length: 89 bytes
Concurrency Level: 1
Time taken for tests: 24.581249 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 15300000 bytes
HTML transferred: 8900000 bytes
Requests per second: 4068.14 [#/sec] (mean)
Time per request: 0.246 [ms] (mean)
Time per request: 0.246 [ms] (mean, across all concurrent requests)
Transfer rate: 607.82 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 0 0 0.2 0 58
Waiting: 0 0 0.2 0 58
Total: 0 0 0.2 0 58
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 0
99% 0
100% 58 (longest request)
8 core - ab test 10x concurrency 100k requests
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /
Document Length: 89 bytes
Concurrency Level: 10
Time taken for tests: 11.864840 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 15300000 bytes
HTML transferred: 8900000 bytes
Requests per second: 8428.26 [#/sec] (mean)
Time per request: 1.186 [ms] (mean)
Time per request: 0.119 [ms] (mean, across all concurrent requests)
Transfer rate: 1259.27 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 7
Processing: 0 0 0.9 1 8
Waiting: 0 0 0.1 0 7
Total: 0 0 0.9 1 8
WARNING: The median and mean for the processing time are not within a normal deviation
These results are probably not that reliable.
WARNING: The median and mean for the total time are not within a normal deviation
These results are probably not that reliable.
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 1
95% 1
98% 1
99% 1
100% 8 (longest request)
8 core - ab test 100x concurrency
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /
Document Length: 89 bytes
Concurrency Level: 100
Time taken for tests: 11.915829 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 15306273 bytes
HTML transferred: 8903649 bytes
Requests per second: 8392.20 [#/sec] (mean)
Time per request: 11.916 [ms] (mean)
Time per request: 0.119 [ms] (mean, across all concurrent requests)
Transfer rate: 1254.38 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 4 95.0 1 3014
Processing: 1 7 3.2 7 226
Waiting: 0 2 2.4 3 224
Total: 1 11 95.1 8 3025
Percentage of the requests served within a certain time (ms)
50% 8
66% 10
75% 11
80% 12
90% 13
95% 15
98% 16
99% 17
100% 3025 (longest request)
8 core - ab test 1000x concurrency
Vert.X crashes.
Raw
Summary
NodeJS 1 core 100k requests
Concurrency 1x - 3821.89 [requests/sec] Test Time: 26.165080 seconds
Concurrency 10x - 5389.03 [requests/sec] Test Time: 18.556226 seconds
Concurrency 100x - 5482.68 [requests/sec] Test Time: 18.239243 seconds
Concurrency 1000x- 5617.01 [requests/sec] Test Time: 17.803068 seconds
NodeJS 8 core 100k requests
Concurrency 1x - 3613.61 [requests/sec] Test Time: 27.673151 seconds
Concurrency 10x - 11624.60 [requests/sec] Test Time: 8.602445 seconds
Concurrency 100x - 10991.01 [requests/sec] Test Time: 9.98349 seconds
Concurrency 1000x- 10171.14 [requests/sec] Test Time: 9.831740 seconds
Vert.x 8 core 100k requests
Concurrency 1x - 4068.14 [requests/sec] Test Time: 24.581249 seconds
Concurrency 10x - 8428.26 [requests/sec] Test Time: 11.864840 seconds
Concurrency 100x - 8392.20 [requests/sec] Test Time: 11.915829 seconds
Concurrency 1000x- VERT.X CRASHES SO THIS TEST WON'T RUN
@courtneycouch
Copy link
Author

Oops wrong instance type listed. I meant I ran on this instance:

High-Memory Quadruple Extra Large Instance

68.4 GB of memory
26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each)
1690 GB of instance storage
64-bit platform
I/O Performance: High
API name: m2.4xlarge

@tsiura
Copy link

tsiura commented Jul 31, 2012

I don't believe in this ... i run vert.x test on my dual-core machine with 2gb ram and with loaded gnome interface and some programs with more than 10k reqs/sec!

ab -c 100 -n 100000 http://localhost:8082/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests

Server Software:

Server Hostname: localhost
Server Port: 8082

Document Path: /
Document Length: 76 bytes

Concurrency Level: 100
Time taken for tests: 9.541 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 15500000 bytes
HTML transferred: 7600000 bytes
Requests per second: 10480.67 #/sec
Time per request: 9.541 ms
Time per request: 0.095 ms
Transfer rate: 1586.43 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 4 60.7 0 1002
Processing: 0 5 7.9 4 659
Waiting: 0 5 7.9 4 658
Total: 0 9 62.3 4 1225

Percentage of the requests served within a certain time (ms)
50% 4
66% 5
75% 6
80% 7
90% 8
95% 10
98% 13
99% 15
100% 1225 (longest request)

@courtneycouch
Copy link
Author

It depends what those requests are doing. The test above involves file IO. It is an extension of this: https://gist.github.com/2652991

Simply posting that you can get much larger numbers of requests through vert.x is sort of a nonsense statement. Of course you can. I'm sure someone can push 30k requests through it. The point of this isn't to say "how many requests can you get through" but to compare the same activity on both platforms to see the performance comparison.

I can easily push 30k requests through nodejs on less hardware, far more than this test is showing. Again this is more of a file io test to compare to the bs vert.x microbenchmarks that came out a while back.

Your results are meaningless without an explanation of what you are testing exactly and some base comparison with something else doing the same workload.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment