Kafka performance benchmark summary

kafka performance benchmark

Kafka benchmarks summary prepared from:
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
https://gist.github.com/jkreps/c7ddb4041ef62a900e6c

I had six machines each has the following specs

Intel Xeon 2.5 GHz processor with six cores
Six 7200 RPM SATA drives
32GB of RAM

3 machines are used for kafka
3 machines are used for zookeeper and generating load


3 nodes, no replication, topic with 6 partitions, small messages 100bytes, single producer
8.2 lakh/sec
78 mb/s
We don't need this one.

3 nodes, 3 replica, small messages 100bytes, single ack, batch size = 8196
7.8 lakh/sec
75 mb/s

3 nodes, 3 replica, small messages 100bytes, majority ack, batch size is 64000 i.e produer is using 64k*6*100 = 36MB buffer.
4.2 lakh/sec
40 mb/sec
This is the configuration we want if we don't want to loose any data. So I am only interested in this.
We use RAID 1, 7200 rpm hdd disk which has around 100 mb/sec sequential write.
But in RAID 1, write performance degrades.
So achieving 40-50 mb/sec is good enough.
If node is using 1Gbps network, 125 MB/s is the maximum transfer rate from producer to single broker node.
We can do this load test with batch size of 64, 124, 256, 512, 1024 and see the results.
And we can do this test for different message sizes 100b, 200b, 500b, 1kb along with different batch sizes.


3 nodes, 3 replica, small messages 100bytes, single ack, 3 producer
20.2 lakh/sec
193 mb/s


3x replicated data, single thread consumer, 6 partitions
9 lakh/s
89 mb/s

3x replicated data, 3 thread consumer, 6 partitions
26  lakh/s
249 mb/s

3x replication, single ack, one producer and one consumer
7.9 lakh/sec
75 mb/sec

end to end latency
2 ms (median)
3 ms (99th percentile)
14 ms (99.9th percentile)


In kafka performance doesn't degrades with data size.

With higher message size mess/s will decrease but mb/sec may improve. With small message size batch size needs to be higher and with bigger message size batch size needs to be lower.
https://content.linkedin.com/content/dam/engineering/en-us/blog/migrated/size_vs_record_throughput.png
https://content.linkedin.com/content/dam/engineering/en-us/blog/migrated/size_vs_mb_throughput.png

We are using dual quad-core Intel Xeon machines with 24GB of memory.
You need sufficient memory to buffer active readers and writers. You can do a back-of-the-envelope estimate of memory needs by assuming you want to be able to buffer for 30 seconds and compute your memory need as write_throughput*30.