DeathBorn
/ Kafka performance tuning
Created
March 26, 2020 08:07
— forked from idris75/Kafka performance tuning
kafka producer - consumer - broker tuning
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 1.Producer | |
| 1.request.required.acks=[0,1,all/-1] 0 no acknowledgement but ver fast, 1 acknowledged after leader commits, all acknowledged after replicated | |
| 2.use Async producer - use callback for the acknowledgement, using property producer.type=1 | |
| 3.Batching data - send multiple messages together. | |
| batch.num.messages | |
| queue.buffer.max.ms | |
| 4.Compression for Large files - gzip, snappy supported | |
| very large files can be stored in shared location and just the file path can be logged by the kafka producer. | |