-
-
Save nohupped/870f28e5ea0c935e261ccbdb43ac0c76 to your computer and use it in GitHub Desktop.
How to alter kafka partitions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# create a topic with specified partitons | |
kafka-topics --bootstrap-server xx.xx.xx.xx:39092 --create --partitions 2 --topic test-partitions | |
kafka-topics --bootstrap-server xx.xx.xx.xx:39092 --describe --topic test-partitions | |
# create a topic with default partitions | |
kafka-topics --bootstrap-server xx.xx.xx.xx:39092 --create --topic test-def-partitions | |
kafka-topics --bootstrap-server xx.xx.xx.xx:39092 --describe --topic test-def-partitions | |
# alter the number of partitions of an existing topic | |
kafka-topics --bootstrap-server xx.xx.xx.xx:39092 --alter --topic test-def-partitions --partitions 3 | |
kafka-topics --bootstrap-server xx.xx.xx.xx:39092 --describe --topic test-def-partitions | |
# Topic: test-def-partitions PartitionCount: 3 ReplicationFactor: 1 Configs: | |
# Topic: test-def-partitions Partition: 0 Leader: 1 Replicas: 1 Isr: 1 Offline: | |
# Topic: test-def-partitions Partition: 1 Leader: 1 Replicas: 1 Isr: 1 Offline: | |
# Topic: test-def-partitions Partition: 2 Leader: 1 Replicas: 1 Isr: 1 Offline: | |
# check the number of message in a topic | |
kafka-run-class kafka.tools.GetOffsetShell --broker-list 20.150.137.17:39092 --topic test-partitions | |
# create a topic mytopic1 with 2 partitions | |
kafka-topics --bootstrap-server 20.150.137.17:39092 --create --partitions 2 --topic mytopic1 | |
# start producing message for the topics mytopic1 | |
kafka-console-producer \ | |
--bootstrap-server 20.150.137.17:39092 \ | |
--topic mytopic1 \ | |
--property parse.key=true \ | |
--property key.separator=, | |
# enter the following messages: | |
a,{"column1": "1"} | |
b,{"column1": "1"} | |
c,{"column1": "1"} | |
d,{"column1": "1"} | |
a,{"column1": "2"} | |
b,{"column1": "2"} | |
c,{"column1": "2"} | |
d,{"column1": "2"} | |
a,{"column1": "3"} | |
b,{"column1": "3"} | |
c,{"column1": "3"} | |
d,{"column1": "3"} | |
# see the messages in the topic mytopic1 and partition 0 | |
kafka-console-consumer \ | |
--bootstrap-server 20.150.137.17:39092 \ | |
--topic mytopic1 \ | |
--property print.key=true \ | |
--property key.separator=, \ | |
--partition 0 \ | |
--from-beginning | |
# see the messages in the topic mytopic1 and partition 1 | |
kafka-console-consumer \ | |
--bootstrap-server 20.150.137.17:39092 \ | |
--topic mytopic1 \ | |
--property print.key=true \ | |
--property key.separator=, \ | |
--partition 1 \ | |
--from-beginning | |
# show the number of messages in echo of the partitions in the topic mytopic1 | |
# https://stackoverflow.com/questions/28579948/java-how-to-get-number-of-messages-in-a-topic-in-apache-kafka | |
kafka-run-class kafka.tools.GetOffsetShell \ | |
--broker-list 20.150.137.17:39092 --topic mytopic1 | |
mytopic1:0:9 | |
mytopic1:1:3 | |
kafka-run-class kafka.admin.ConsumerGroupCommand \ | |
--all-groups \ | |
--bootstrap-server 20.150.137.17:39092 \ | |
--describe | grep mytopic1 | |
Consumer group '_confluent-controlcenter-6-2-0-lastProduceTimeConsumer' has no active members. | |
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID | |
_confluent-controlcenter-6-2-0-lastProduceTimeConsumer mytopic1 1 3 3 0 - - - | |
_confluent-controlcenter-6-2-0-lastProduceTimeConsumer mytopic1 0 9 9 0 - - - | |
# ###########ConsumerOffsetChecker################# has been deprecated. Going forward, | |
# please use kafka-consumer-groups.sh (kafka.admin.ConsumerGroupCommand) for this functionality. | |
# kafka-run-class kafka.admin.ConsumerOffsetChecker --topic mytopic1 --broker-list 20.150.137.17:39092 --all-groups |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment