Kafka docker-compose and kafkacat

kafka-docker.md

A working setup for local kafka in docker-compose, using kafkacat as a CLI client

Setup Docker compose

Add these sections to your docker compose to bring kafka, zookeeper and required volumes up

  zookeeper:
    image: bitnami/zookeeper:latest
    ports:
      - "2181:2181"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
    volumes:
      - zookeeper_data:/bitnami/zookeeper


  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - "9092:9092"
    environment:
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
      - KAFKA_CFG_LISTENERS=INTERNAL://kafka:29092,CONTROLLER://:9093,EXTERNAL://:9092,OUTSIDE://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka:29092,EXTERNAL://localhost:9092,OUTSIDE://localhost:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
    restart:
      always
    volumes:
      - kafka_data:/bitnami/kafka
    depends_on:
      - zookeeper

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

Useful kafkacat commands

To explore existing topics:

kcat -b localhost:9092 -L | grep topic

To consume from topic <topic>:

kcat -b 127.0.0.1:9092 -t <topic> -K############

This will display the key/value pairs of each message, separated by ############

To produce a binary payload to a topic, create a file containing the binary content, then produce, in this case we produce with a constant key

kcat -b 127.0.0.1:9092 -t <topic>  -T -P -k default-key  proto-msgs/<binary-file>