Useful Commands

Raw

jar.md

Jar Files

Extract a jar: jar xf <jar-file>

View contents: jar tf <jar-file>

Create:

All files in current folder: jar cf <new-file-name> .
Custom: jar cf <new-file-name> [input file(s)]

Raw

kafka.md

Kafka

Zookeeper Start: bin/zookeeper-server-start.sh config/zookeeper.properties &

Kafka Server Start: bin/kafka-server-start.sh config/server.properties &

Create Topic: bin/kafka-create-topic.sh --zookeeper localhost:2181 --topic test11

Start Producer: bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test11

Start Consumer: bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test11

Benchmark

Command: bin/kafka-producer-perf-test.sh --broker-list=localhost:9092 --messages 10000000 --topic perf --threads 10 --message-size 1000 --batch-size 200 --compression-codec 1

start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec

2014-11-26 14:39:13:801, 2014-11-26 14:42:27:464, 1, 1000, 200, 9536.74, 49.2440, 10000000, 51636.0895

2014-12-05 13:43:43:321, 2014-12-05 13:46:29:254, 1, 1000, 200, 9536.74, 57.4735, 10000000, 60265.2878

Kafka Performance Notes

Topic with 3 partitions:

Speed with 10 threads: 71 MB/s

Speed with 1 thread: 27 MB/s

Raw

ports.md

Ports

sudo lsof -i :2181 lists lists of files using port 2181

Hadoop

Hadoop's namenode has gone into safemode. Leave it: bin/hadoop dfsadmin -safemode leave

Git

Google Doc

ssh key

ssh-keygen -t rsa

sed

Get broker-list from zookeeper

echo 'kafka-kamal-1:2181,kafka-kamal-2:2181,kafka-kamal-3:2181,kafka-kamal-4:2181' | sed 's/:2181/:9092/g'

Raw

time.md

Timestamps

date gives Tue Nov 18 20:04:31 IST 2014

date +%s gives 1416321343

date -r 1416321343 gives Tue Nov 18 20:05:43 IST 2014

Raw

Useful.md

SBT

echo "deb http://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list

sudo apt-get update

sudo apt-get install sbt

Java