Skip to content

Instantly share code, notes, and snippets.

View manuzhang's full-sized avatar
🧒
Working from home

Manu Zhang manuzhang

🧒
Working from home
262 followers · 412 following
View GitHub Profile
@manuzhang
manuzhang / run_benchmark.sh
Last active August 29, 2015 14:18
# command to run storm-benchmark over Gearpump
bin/gear app -jar stormbench/storm-benchmark-0.1.0-jar-with-dependencies.jar org.apache.gearpump.experiments.storm.StormRunner -storm_topology storm.benchmark.tools.Runner -storm_args storm.benchmark.benchmarks.SOL -storm_config /root/gearpump-pack-0.3.3-SNAPSHOT/stormbench/sol.yaml -master 192.168.1.71:3000
# command to run storm-benchmark over Storm
stormbench/stormbench -jar stormbench/storm-benchmark-0.1.0-jar-with-dependencies.jar -storm /root/apache-storm-0.9.3/bin/storm -conf stormbench/sol.yaml storm.benchmark.tools.Runner storm.benchmark.benchmarks.SOL
@manuzhang
manuzhang / sol_storm_acker_off.yaml
Last active August 29, 2015 14:18
# metrics configurations
metrics.enabled: true
metrics.poll: 60000 # 60 secs
metrics.time: 900000 # 15 mins
metrics.path: "reports"
# topology configurations
topology.workers: 4
topology.acker.executors: 0
topology.max.spout.pending: 200
@manuzhang
manuzhang / sol_storm_acker_off.csv
Last active August 29, 2015 14:18
time(s) total_slots used_slots workers tasks executors transferred (messages) throughput (messages/s) throughput (MB/s) spout_throughput (MB/s) spout_executors spout_transferred (messages) spout_acked (messages) spout_throughput (messages/s) spout_avg_complete_latency(ms) spout_max_complete_latency(ms)
60 16 4 4 64 64 100149200 1668958 166.0 166.000 32 100149200 0 1668958 0.0 0.0
120 16 4 4 64 64 12551240 208801 20.0 20.000 32 12551240 0 208801 0.0 0.0
180 16 4 4 64 64 12042100 200441 20.0 20.000 32 12042100 0 200441 0.0 0.0
240 16 4 4 64 64 12110060 201646 20.0 20.000 32 12110060 0 201646 0.0 0.0
300 16 4 4 64 64 12380240 206151 20.0 20.000 32 12380240 0 206151 0.0 0.0
360 16 4 4 64 64 11845460 197253 19.0 19.000 32 11845460 0 197253 0.0 0.0
420 16 4 4 64 64 12341160 205566 20.0 20.000 32 12341160 0 205566 0.0 0.0
480 16 4 4 64 64 12248660 203977 20.0 20.000 32 12248660 0 203977 0.0 0.0
540 16 4 4 64 64 11745900 195657 19.0 19.000 32 11745900 0 195657 0.0 0.0
@manuzhang
manuzhang / sol_storm_acker_on.yaml
Last active August 29, 2015 14:18
# metrics configurations
metrics.enabled: true
metrics.poll: 60000 # 60 secs
metrics.time: 900000 # 15 mins
metrics.path: "reports"
# topology configurations
topology.workers: 4
topology.acker.executors: 4
topology.max.spout.pending: 200
@manuzhang
manuzhang / sol_storm_acker_on.csv
Last active August 29, 2015 14:18
We can make this file beautiful and searchable if this error is corrected: It looks like row 10 should actually have 16 columns, instead of 12 in line 9.
time(s),total_slots,used_slots,workers,tasks,executors,transferred (messages),throughput (messages/s),throughput (MB/s),spout_throughput (MB/s),spout_executors,spout_transferred (messages),spout_acked (messages),spout_throughput (messages/s),spout_avg_complete_latency(ms),spout_max_complete_latency(ms)
60,16,4,4,68,68,12581640,209669,20.0,20.000,32,12581640,12568820,209669,26.4,28.1
120,16,4,4,68,68,15556400,258811,25.0,25.000,32,15556400,28122640,258811,25.6,27.4
180,16,4,4,68,68,15141200,252046,25.0,25.000,32,15141200,43243080,252046,25.3,27.3
240,16,4,4,68,68,15685280,261168,26.0,26.000,32,15685280,58923020,261168,25.2,26.8
300,16,4,4,68,68,15406060,256588,25.0,25.000,32,15406060,74317520,256588,25.1,26.6
360,16,4,4,68,68,14788500,246306,24.0,24.000,32,14788500,89099380,246306,25.2,26.6
420,16,4,4,68,68,14489620,241356,24.0,24.000,32,14489620,103584100,241356,25.4,26.6
480,16,4,4,68,68,14862080,247569,24.0,24.000,32,14862080,118429540,247569,25.5,26.8
540,16,4,4,68,68,14920480,248521,24.0,24.000,32,1492048
@manuzhang
manuzhang / add_window_metrics.patch
Created February 24, 2017 02:16
Patch of adding metrics to WindowOperator
Index: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java (date 1487603477000)
+++ flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java (revision )
@@ -38,6 +38,8 @@
import org.apache.flink.core.fs.FSDataInputStream;
import org.apache.flink.core.memory.DataInputView;
@manuzhang
manuzhang / simple_barrier.py
Last active March 27, 2020 07:12 — forked from yaroslavvb/simple_barrier.py
TensorFlow in-graph replication example
"""
This example is adapted from https://gist.github.com/yaroslavvb/ef407a599f0f549f62d91c3a00dcfb6c
Example of barrier implementation using TensorFlow shared variables.
All workers synchronize on barrier, copy global parameters to local versions
and increment global parameter variable asynchronously. Should see something
like this:
python simple_barrier.py --wk "node13-1:21393,node13-1:21395"
Creating session
@manuzhang
manuzhang / demo.py
Created April 27, 2017 04:51
demo script for https://github.com/Intel-bigdata/TensorFlowOnYARN
import json
import subprocess
import sys
# Example:
# python demo.py "bin/ydl-tf" "launch" "examples/between-graph/mnist_feed.py"
def main():
args = sys.argv[1:]
output = subprocess.check_output([args[0], args[1]])
@manuzhang
manuzhang / MurmurHashEncode.scala
Last active September 15, 2017 02:35
Encode with Murmur Hash
import java.nio.{ByteBuffer, ByteOrder}
// "com.google.guava" % "guava" % "16.0.1"
import com.google.common.hash.Hashing
object MurmurHash {
private val seed = 0x3c074a61
def encode(prefix: Int, value: Long): Long = {
val pb = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(prefix).array()
@manuzhang
manuzhang / spark_catalyst_time.md
Last active August 13, 2018 13:17
Rule Nano Time
org.apache.spark.sql.catalyst.analysis.Analyzer$FixNullability 489262230
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGroupingAnalytics 243030776
org.apache.spark.sql.catalyst.analysis.TypeCoercion$PropagateTypes 143141555
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveDeserializer 97690381
org.apache.spark.sql.catalyst.analysis.ResolveCreateNamedStruct 87845664
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveWindowFrame 85098172
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveWindowOrder 83967566
org.apache.spark.sql.catalyst.analysis.Analyzer$ExtractWindowExpressions 63928074