andfadeev/kafka-consume-from-timestamp.md

Created June 26, 2018 09:29

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/andfadeev/879f456059bda42b355508648dbac2a4.js"></script>
Save andfadeev/879f456059bda42b355508648dbac2a4 to your computer and use it in GitHub Desktop.

Consume kafka topic from timestamp with kafka-console-consumer.sh

Raw

Sometime it is usefull to consume kafka topic starting from specific timestamp (ex. for debug and faster search specific records).

(Optional) Start docker container with kafka bin

docker run -it wurstmeister/kafka bash

Choose unique group-id and reset offsets

$KAFKA_HOME/bin/kafka-consumer-groups.sh \
 --bootstrap-server your-kafka-url:9092 \
 --group some-unique-group-id \
 --reset-offsets \
 --topic some-topic  \
 --to-datetime "YYYY-MM-DDTHH:MM:SS.sss" \
 --execute

Start consumer with group-id

$KAFKA_HOME/bin/kafka-console-consumer.sh \
 --bootstrap-server your-kafka-url:9092 \
 --topic some-topic \
 --group some-unique-group-id

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment