1 leader is elected per partition. Many followers. Followers calculate the offset. If leader crashes, a new leader is elected. Non replicated messages are lost.
Producers & consumers deal with the leader.
Producer request topology metadata. For 1 topic with 3 partition, we can have 3 leaders (to loadbalance).
Broker cluster are consumed by groups. Each group can listen to all partitions. In a group, there could only be 1 consumer for 1 partition.
A consumer can be low-level. It deals with wire protocol, coordinates & offsets. Or it can be high-level. Then all the orchestration is already done.
Consumer failover: traffic is rebalanced to another consumer (some client libraries does not do it for you).
- ISR: In-sync replica
- Topic
- Partition
- Producer
- Broker
- Consumer
Start configuration service
/opt/apache/kafka_2.11-0.10.1.1$ bin/zookeeper-server-start.sh config/zookeeper.properties Start the broker
/opt/apache/kafka_2.11-0.10.1.1$ sudo bin/kafka-server-start.sh config/server.properties Lists the topics
/opt/apache/kafka_2.11-0.10.1.1$ bin/kafka-topics.sh --list --zookeeper localhost:2181echo Hello world | bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Hello-Kafkabin/kafka-console-consumer.sh --zookeeper localhost:2181 —topic Hello-Kafka --from-beginning