LearnKafka.md

Kafka

Overview

Originally developed at LinkedIn. Written in Java. Client module supports many languages.

Kafka = A Distribued Commit Log

Concept: https://www.youtube.com/watch?v=U4y2R3v9tlY

Example: https://www.youtube.com/watch?v=UEg40Te8pnE

Guarantees

1. Ordering:
   1. Messages sent by a producer to a topic and a partition are appended in the same order
   2. A consumer instance gets the messages in the same order as they are produced
2. Fault Tolerance: A topic with replication factor N, tolerates upto N-1 server failures

Components

                              +---------+
                              |ZooKeeper|
                              +--+---^--+
                                 |   |
                                 |   |
                +----------------v---+---------------+
                |            Kafka Cluster           |
                | +------+                  +------+ |
+--------+      | |Broker|                  |Broker| |   +--------------+
|Producer+------> +--+---+                  +--^---+ +--->ConsumerGroup1|
+--------+      |    |                         |     |   +--------------+
                |    |  +------------------+   |     |
+--------+      |    +--> Topic Partition1 +---+     |   +--------------+
|Producer+------>    |  +------------------+   |     +--->ConsumerGroup2|
+--------+      |    |                         |     |   +--------------+
                |    |  +------------------+   |     |
                |    +--> Topic Partition2 +---+     |
                |       +------------------+         |
                |                                    |
                +------------------------------------+

Configurations

Producers

• Async (No Guarantee)
• Committed to Leader
• Committed to Leader & Quotum

Consumer

• At least once (Default)
• At most once
• Effectively once
• Exactly once (Maybe)

Communication Pattern

1. Publish-subscribe
2. Queue system

API

class Producer {
    public void send(ProducerData<K, V> producerData);
}

class SimpleConsumer {
    public ByteBufferMessageSet fetch(FetchRequest request);
    
    public log[] getOffsetsBefore(string topic, int partition, ...)
}