Run Kafka Container

run-kafka-container.md

Start Kafka service

The following commands will start a container with Kafka and Zookeeper running on mapped ports 2181 (Zookeeper) and 9092 (Kafka).

docker pull spotify/kafka
docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=kafka --env ADVERTISED_PORT=9092 --name kafka spotify/kafka

Why Spotify?

ADVERTISTED_HOST was set to kafka, which will allow other containers to be able to run Producers and Consumers.

Setting ADVERTISED_HOST to localhost, 127.0.0.1, or 0.0.0.0 will work great only if Producers and Consumers are started within the kafka container itself, or if you are using DockerForMac (like me) and you want to run Producers and Consumers from OSX. These are far less interesting use cases though, so we'll start Producers and Consumers from other containers.

We need to use an IP address or hostname in order for the kafka service to be reachable from another container. IP address is not known before the container is started, so we have to choose a hostname, and I chose kafka in this example.

Create a topic

docker exec kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

output:

Created topic "test".

List topics

docker exec kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --list --zookeeper localhost:2181

output:

test

Start a producer (in a new terminal window)

This command will run an unnamed instance of spotify/kafka linked to the kafka service, start a producer, and wait for newline-delimited input until you quit (which destroys the container):

docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list kafka:9092 --topic test

Start a consumer (in a new terminal window)

This command will start an unnamed instance of spotify/kafka linked to the kafka service, start a consumer, display existing messages from the test topic, and wait for new messages until you quit (which destroys the container):

docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --from-beginning

Send messages

Send some newline-delimited messages in the Producer terminal window. The messages appear in the Consumer terminal window.

Thanks for sharing this one it really helps a lot.
One thing I noticed is when running consumer, my terminal window was flood with this message
WARN Error while fetching metadata with correlation id 6 : {testing=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

and typing something in producer window also emit this kind of message.

I'm able to resolve the issue by adding --hostname when starting the kafka service. [stackoverflow]

First of all Thanks! This got me started quickly.

I was getting lot of error like this:
[2017-12-21 04:46:22,224] WARN Error while fetching metadata with correlation id 95 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

This was solved by adding "-h kafka" when starting the container.

docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=kafka --env ADVERTISED_PORT=9092 --name kafka -h kafka spotify/kafka

I am geting below error.
I tried kafka also and localhost also but same result.
docker@default:~$ docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

abcddd
[2018-01-18 11:48:19,683] ERROR Error when sending message to topic test with key: null, value: 0 by
tes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

Can anybody please help to resolve this issue.

@Rohit2387
Hey,

I had this error also,
in my case this error occurs if you run docker container in composer in network,
you should run in terminal command docker inspect kafka | grep IPAddress, find ip below part network
and run consumer/producer like this
docker run -it --rm --net my_net --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list ip_from_inspect:9092 --topic test

Works fine, thanks a lot

Hi,

Thanks for pointing to this simple kafka implementation, it is really helpful to kafka beginners like me. Can you please point me some sample application of .net client using which I can produce and consume kafka messages from console application.

@Rohit2387

i found the issue, 9092 was not my port. For me listening port was 6667 (you can check this at kafka.properties file)
Add the following
listeners=PLAINTXT://0.0.0.0:9092
listeners=PLAINTEXT://localhost.localdomain:9092

@danswater

I had the same issue, and it works for me by using the commands like this (ie. using ip address instead of "kafka")
docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=192.168.99.100 --env ADVERTISED_PORT=9092 --name kafka spotify/kafka

docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list 192.168.99.100:9092 --topic test

docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-consumer.sh --bootstrap-server 192.168.99.100:9092 --topic test --from-beginning

Using you commands i have this output in producer and consumer

[2018-09-19 16:57:11,688] WARN Error while fetching metadata with correlation id 245 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-09-19 16:57:11,793] WARN Error while fetching metadata with correlation id 246 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

Using you commands i have this output in producer and consumer

[2018-09-19 16:57:11,688] WARN Error while fetching metadata with correlation id 245 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-09-19 16:57:11,793] WARN Error while fetching metadata with correlation id 246 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=kafka --env ADVERTISED_PORT=9092 --hostname kafka --name kafka spotify/kafka

Great article! Thank you!

Thank you! I want to add some notes here:

1. **Topic description: ** docker exec kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --describe --topic ${topicName} --zookeeper localhost:2181
2. **If you faced with "unknown hostName", try this command to start Kafka: ** docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=${your server's IP} --env ADVERTISED_PORT=9092 --hostname kafka --name kafka spotify/kafka