Skip to content

Instantly share code, notes, and snippets.

@tclh123
Created April 30, 2021 08:53
Show Gist options
  • Save tclh123/990496876bd09663da52f1a19a7d5904 to your computer and use it in GitHub Desktop.
Save tclh123/990496876bd09663da52f1a19a7d5904 to your computer and use it in GitHub Desktop.
Change replica factor for kafka topics. You need to regenerate and assign all partitions after that if there're more than 1.
#!/bin/bash
topics=__consumer_offsets
new_servers=1,2
while read -r line; do lines+=("$line"); done <<<"$topics"
echo '{"version":1,
"partitions":[' > topic-expand-plan.json
for t in $topics; do
if [ "${t}" == "${lines[-1]}" ]; then
echo " {\"topic\":\"${t}\",\"partition\":0,\"replicas\":[$new_servers]}" >> topic-expand-plan.json
else
echo " {\"topic\":\"${t}\",\"partition\":0,\"replicas\":[$new_servers]}," >> topic-expand-plan.json
fi
done
echo ' ]
}' >> topic-expand-plan.json
~/projects/3/kafka_2.12-2.5.0/bin/kafka-reassign-partitions.sh \
--zookeeper "z-1.hansonmessagelogs1.k3r2yj.c4.kafka.eu-west-1.amazonaws.com:2181,z-2.hansonmessagelogs1.k3r2yj.c4.kafka.eu-west-1.amazonaws.com:2181,z-3.hansonmessagelogs1.k3r2yj.c4.kafka.eu-west-1.amazonaws.com:2181" \
--reassignment-json-file topic-expand-plan.json --execute
for t in $topics; do
~/projects/3/kafka_2.12-2.5.0/bin/kafka-topics.sh \
--zookeeper "z-1.hansonmessagelogs1.k3r2yj.c4.kafka.eu-west-1.amazonaws.com:2181,z-2.hansonmessagelogs1.k3r2yj.c4.kafka.eu-west-1.amazonaws.com:2181,z-3.hansonmessagelogs1.k3r2yj.c4.kafka.eu-west-1.amazonaws.com:2181" \
--describe --topic ${t}
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment