Gnat Request Benchmark

README.md

I wanted to run another round of performance benchmarks for gnat to see how it's request throughput has changed with the introduction of the ConsumerSupervisor which handles things like processing each request in its own supervised process.

I used a CPU-optimized digital ocean droplet with 16 cores, gnatsd 1.3.0, erlang 21.2.2 and elixir 1.8.0.rc0 You can read the setup instructions below for more details and the results_by_concurrency.md contains details about a lot of different runs.

I'm trying to measure the overhead in the system, so the requests are random byte strings that just get echoed back with processing. The measurements use byte strings of 4 bytes up to 1024 bytes.

TL/DR; You can do 170k+ synchronous requests/sec of small messages or 192MB+/sec of 1kb messages

results_by_concurrency.md

Gnat Running on Client Machine

client conns	client requesters	server conns	server handlers	message size	req/sec	kb/sec	median	90th %
4	512	4	4	16	93379	2918.1	4103	9972
4	512	8	8	16	93439.82188033953	2919.9944337606103	4269	8743
4	2048	8	8	16	97498.7314572397	3046.8353580387407	19507	25478
4	2048	8	16	16	95559.2278357125	2986.2258698660157	19851	25849
4	2048	16	32	16	95202.40097852077	2975.075030578774	20076	25387

Gnat Running on Server Machine

client conns	client requesters	server conns	server handlers	message size	req/sec	kb/sec	median	90th %
4	1024	8	8	16	108550.8953699757	3392.2154803117405	8335	12124
8	1024	8	8	16	140868.9800177902	4402.1556255559435	6035	11156
8	1024	16	16	16	149550.08281116767	4673.44008784899	5484	10432
8	2048	16	16	16	156817.07885004688	4900.533714063965	10882	18424
16	2048	16	16	16	34329.764239170705	1072.8051324740845	58830	68563
12	2048	16	16	16	50866.57721633351	1589.5805380104223	47780	55717
10	2048	16	16	16	100257.3843993929	3133.0432624810283	14339	43591
8	2048	16	16	16	153525.29842791744	4797.66557587242	11048	18747
6	1024	16	16	16	144757.3148564192	4523.6660892631	5760	9701
8	1024	16	16	16	146443.59617277386	4576.362380399183	5549	10740
8	4096	16	16	16	168454.1685600282	5264.192767500881	20770	30994
8	4096	16	16	256	148215.92498334651	74107.96249167326	24703	35386
8	4096	16	16	1024	98581.15579329274	197162.31158658548	39292	59472
8	4096	16	16	4	170606.2436220505	1332.8612782972696	20249	31148
8	4096	8	8	4	168135.7403836393	1313.560471747182	21324	33085
8	6000	8	8	4	165030.80341203389	1289.3031516565147	31588	45289
8	6000	4	4	4	129725.25596144305	1013.4785621987738	36778	93851

Median and 90th % are reported in microseconds of latency to send request and receive parsed response

setup_instructions.md

# apt-get install build-essential unzip libssl-dev automake autoconf libncurses5-dev 
# git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.6.0
# echo -e '\n. $HOME/.asdf/asdf.sh' >> ~/.bashrc
# echo -e '\n. $HOME/.asdf/completions/asdf.bash' >> ~/.bashrc
# source ~/.bashrc
# asdf plugin-add erlang
# asdf plugin-add elixir
# asdf install erlang 21.2.2
# asdf global erlang 21.2.2
# asdf install elixir 1.8.0-rc.0
# asdf global elixir 1.8.0-rc.0
# wget https://github.com/nats-io/gnatsd/releases/download/v1.3.0/gnatsd-v1.3.0-linux-amd64.zip
# unzip gnatsd-v1.3.0-linux-amd64.zip
# ./gnatsd-v1.3.0-linux-amd64/gnatsd &
# git clone https://github.com/mmmries/gnat.git
# cd gnat
# mix do deps.get, compile
# mix run bench/server.exs # on the server machine
# mix run bench/client.exs # on the client machine

straight_line_benchmarks.txt

root@ubuntu-c-16-sfo2-01:~/gnat# mix run bench/parse.exs
Operating System: Linux"
CPU Information: Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
Number of Available Cores: 16
Available memory: 31.42 GB
Elixir 1.8.0
Erlang 21.2.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 0 μs
parallel: 1
inputs: 1024 byte, 128 byte, 16 byte
Estimated total run time: 36 s


Benchmarking parsec with input 1024 byte...
Benchmarking parsec with input 128 byte...
Benchmarking parsec with input 16 byte...

##### With input 1024 byte #####
Name             ips        average  deviation         median         99th %
parsec      438.92 K        2.28 μs  ±1748.29%           2 μs           4 μs

##### With input 128 byte #####
Name             ips        average  deviation         median         99th %
parsec      606.19 K        1.65 μs  ±2093.53%           1 μs           3 μs

##### With input 16 byte #####
Name             ips        average  deviation         median         99th %
parsec      639.57 K        1.56 μs  ±2452.74%           1 μs           3 μs
root@ubuntu-c-16-sfo2-01:~/gnat# mix run bench/publish.exs
Operating System: Linux"
CPU Information: Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
Number of Available Cores: 16
Available memory: 31.42 GB
Elixir 1.8.0
Erlang 21.2.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 0 μs
parallel: 1
inputs: 1024 byte, 128 byte, 16 byte
Estimated total run time: 36 s


Benchmarking pub with input 1024 byte...
Benchmarking pub with input 128 byte...
Benchmarking pub with input 16 byte...

##### With input 1024 byte #####
Name           ips        average  deviation         median         99th %
pub       225.62 K        4.43 μs   ±589.95%           4 μs           9 μs

##### With input 128 byte #####
Name           ips        average  deviation         median         99th %
pub       158.10 K        6.33 μs   ±311.64%           5 μs          11 μs

##### With input 16 byte #####
Name           ips        average  deviation         median         99th %
pub       162.02 K        6.17 μs   ±324.82%           5 μs          12 μs
root@ubuntu-c-16-sfo2-01:~/gnat# mix run bench/request.exs
Operating System: Linux"
CPU Information: Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
Number of Available Cores: 16
Available memory: 31.42 GB
Elixir 1.8.0
Erlang 21.2.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 0 μs
parallel: 1
inputs: 1024 byte, 128 byte, 16 byte
Estimated total run time: 36 s


Benchmarking request with input 1024 byte...
Benchmarking request with input 128 byte...
Benchmarking request with input 16 byte...

##### With input 1024 byte #####
Name              ips        average  deviation         median         99th %
request        8.81 K      113.50 μs    ±38.27%         111 μs         150 μs

##### With input 128 byte #####
Name              ips        average  deviation         median         99th %
request        8.72 K      114.68 μs    ±14.35%         112 μs      154.33 μs

##### With input 16 byte #####
Name              ips        average  deviation         median         99th %
request        8.81 K      113.54 μs    ±15.65%         111 μs         150 μs