Skip to content

Instantly share code, notes, and snippets.

View manuzhang's full-sized avatar
🧒
Working from home

Manu Zhang manuzhang

🧒
Working from home
279 followers · 415 following
View GitHub Profile
@manuzhang
manuzhang / sol_storm_acker_off.csv
Last active August 29, 2015 14:18
time(s) total_slots used_slots workers tasks executors transferred (messages) throughput (messages/s) throughput (MB/s) spout_throughput (MB/s) spout_executors spout_transferred (messages) spout_acked (messages) spout_throughput (messages/s) spout_avg_complete_latency(ms) spout_max_complete_latency(ms)
60 16 4 4 64 64 100149200 1668958 166.0 166.000 32 100149200 0 1668958 0.0 0.0
120 16 4 4 64 64 12551240 208801 20.0 20.000 32 12551240 0 208801 0.0 0.0
180 16 4 4 64 64 12042100 200441 20.0 20.000 32 12042100 0 200441 0.0 0.0
240 16 4 4 64 64 12110060 201646 20.0 20.000 32 12110060 0 201646 0.0 0.0
300 16 4 4 64 64 12380240 206151 20.0 20.000 32 12380240 0 206151 0.0 0.0
360 16 4 4 64 64 11845460 197253 19.0 19.000 32 11845460 0 197253 0.0 0.0
420 16 4 4 64 64 12341160 205566 20.0 20.000 32 12341160 0 205566 0.0 0.0
480 16 4 4 64 64 12248660 203977 20.0 20.000 32 12248660 0 203977 0.0 0.0
540 16 4 4 64 64 11745900 195657 19.0 19.000 32 11745900 0 195657 0.0 0.0
@manuzhang
manuzhang / sol_storm_acker_off.yaml
Last active August 29, 2015 14:18
# metrics configurations
metrics.enabled: true
metrics.poll: 60000 # 60 secs
metrics.time: 900000 # 15 mins
metrics.path: "reports"
# topology configurations
topology.workers: 4
topology.acker.executors: 0
topology.max.spout.pending: 200
@manuzhang
manuzhang / run_benchmark.sh
Last active August 29, 2015 14:18
# command to run storm-benchmark over Gearpump
bin/gear app -jar stormbench/storm-benchmark-0.1.0-jar-with-dependencies.jar org.apache.gearpump.experiments.storm.StormRunner -storm_topology storm.benchmark.tools.Runner -storm_args storm.benchmark.benchmarks.SOL -storm_config /root/gearpump-pack-0.3.3-SNAPSHOT/stormbench/sol.yaml -master 192.168.1.71:3000
# command to run storm-benchmark over Storm
stormbench/stormbench -jar stormbench/storm-benchmark-0.1.0-jar-with-dependencies.jar -storm /root/apache-storm-0.9.3/bin/storm -conf stormbench/sol.yaml storm.benchmark.tools.Runner storm.benchmark.benchmarks.SOL
@manuzhang
manuzhang / sol_gearpump.yaml
Last active August 29, 2015 14:18
# metrics configurations
metrics.enabled: false
metrics.poll: 60000 # 60 secs
metrics.time: 900000 # 15 mins
metrics.path: "reports"
# topology configurations
topology.workers: 4
topology.acker.executors: 0
topology.max.spout.pending: 200
@manuzhang
manuzhang / sbt_china_mirror
Last active March 21, 2020 06:46
sbt China mirror
[repositories]
local
oschina: http://maven.oschina.net/content/groups/public/
oschina-ivy: http://maven.oschina.net/content/groups/public/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
typesafe: http://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
#sonatype-oss-releases
#maven-central
#sonatype-oss-snapshots
@manuzhang
manuzhang / kafka_source_benchmark_commands
Created February 10, 2015 04:46
kafka_source_benchmark_commands
Setup
bin/kafka-topics.sh --create --topic consumer --zookeeper 192.168.1.73:2181/kafka --replication-factor 3 --partitions 6
Produce data
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance consumer 500000000 100 -1 acks=1 bootstrap.servers=192.168.1.73:9092 buffer.memory=67108864 batch.size=8196
Single KafkaSource
bin/gear app -jar examples/gearpump-examples-assembly-0.2.4-SNAPSHOT.jar org.apache.gearpump.streaming.examples.kafka.consumer.KafkaConsumerPerf -master 192.168.1.71:3000 -kafka_stream_producer 1 -runseconds 360
KafkaSource + StreamProcessor
@manuzhang
manuzhang / kill_jps_by_name.sh
Created January 29, 2015 00:47
kill all java processes with the given name
jps | grep ${process_name} | cut -d ' ' -f 1 | xargs kill -9
@manuzhang
manuzhang / benchmark-commands.txt
Last active August 29, 2015 14:10 — forked from jkreps/benchmark-commands.txt
Producer
Setup
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test --partitions 6 --replication-factor 3
Single thread, no replication
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196
@manuzhang
manuzhang / data_cleanse.py
Last active August 29, 2015 14:05
cleanse stock tweets
ticker_dict = {}
company_list = []
exception_list = []
print "read ticker symbol"
with open("./ticker_symbol.tsv") as ticker_symbol:
for line in ticker_symbol:
words = line.split('\t')
ticker_dict[words[0].strip('"').lower()] = words[1]
@manuzhang
manuzhang / hadoop_exception.md
Last active August 29, 2015 14:04
Exception when creating Hadoop FileSystem due to incorrect native hadoop library

The following exception could be thrown when creating Hadoop FileSystem caused by incorrect hadoop native library (configured by LD_LIBRARY_PATH)

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
    at org.apache.hadoop.security.Groups.<init>(Groups.java:55)
    at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182)
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:235)
    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:214)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)

at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)