bin/kafka-topics.sh --zookeeper localhost:2181 --list
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic
bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000
... wait a minute ...
# Calculate the latest and earliest offsets, and provide the total # of messages in the topic (the difference): | |
#!/bin/bash | |
if [[ $# -eq 0 ]] ; then | |
echo 'provide the name of a topic to get totals against' | |
exit 0 | |
fi | |
latest=`bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic $1 --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'` | |
earliest=`bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic $1 --time -2 --offsets 1 | awk -F ":" '{sum2 += $3} END {print sum2}'` | |
total=`expr $latest - $earliest` |
bin/kafka-topics.sh --zookeeper localhost:2181 --list
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic
bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000
... wait a minute ...
#!/bin/bash | |
if [ $# -ne 2 ]; then | |
>&2 echo "usage: $0 <kafka_binary_folder> <bootstrap.server>" | |
>&2 echo | |
>&2 echo "It outputs the list of topics consumed by consumergroup as csv." | |
>&2 echo "With the following structure:" | |
>&2 echo " <consumer>,<topic>" | |
exit 1 | |
fi |
Producer | |
Setup | |
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1 | |
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test --partitions 6 --replication-factor 3 | |
Single thread, no replication | |
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196 |
Assuming that the following environment variables are set:
KAFKA_HOME
where Kafka is installed on local machine (e.g. /opt/kafka
)ZK_HOSTS
identifies running zookeeper ensemble, e.g. ZK_HOSTS=192.168.0.99:2181
KAFKA_BROKERS
identifies running Kafka brokers, e.g. KAFKA_BROKERS=192.168.0.99:9092
Start Zookepper and Kafka servers
UPDATE 2016-12-01: Please refer to the official guide instead of this process.
UPDATE 2014-12-21: RubyGems 1.8.30, 2.0.15 and 2.2.3 have been released. It requires manual installation, please see instructions below.
Hello,
#Container Resource Allocation Options in docker-run You have various options for controlling resources (cpu, memory, disk) in docker. These are principally via the docker-run command options.
##Dynamic CPU Allocation
-c, --cpu-shares=0
CPU shares (relative weight, specify some numeric value which is used to allocate relative cpu share)
##Reserved CPU Allocation
It's a common confusion about terminal colours... Actually we have this:
printf "\x1b[${bg};2;${red};${green};${blue}m\n"
# base requirements | |
sudo apt-get update | |
sudo apt-get install -y software-properties-common | |
sudo apt-get install -y python-setuptools python-dev libffi-dev libssl-dev git sshpass tree | |
sudo easy_install pip | |
sudo -H pip install cryptography | |
# sudo -H pip install ansible==2.1.1.0 if you don't need virtualenv | |
# virtualenv |