Skip to content

Instantly share code, notes, and snippets.

View qix's full-sized avatar

Josh Yudaken qix

33 followers · 2 following
View GitHub Profile
@qix
qix / gist:5a38e4533b25484bc1f7
Created May 20, 2015 00:40
[
{
"dataSchema": {
"dataSource": "allFeedFeatures",
"granularitySpec": {
"queryGranularity": "NONE",
"segmentGranularity": "DAY",
"type": "uniform"
},
"metricsSpec": [
@qix
qix / gist:7b8122539c9b4969c35b
Last active August 29, 2015 14:21

== Simple cluster setup ==

  • One druid broker: druid-broker1
  • One historical node: druid-historical3
  • One realtime node: druid-realtime2

Configuration details at the bottom

== Running against druid-broker1 value occassionally changes by 300000 ==

@qix
qix / gist:074cda2f595e6f5883af
Created July 27, 2015 23:17
2015-07-27 23:16:09 =ERROR REPORT====
** Generic server <0.17476.23> terminating
** Last message in was {'EXIT',<0.17475.23>,{{badmatch,{reply,{riak_kv_w1c_put_reply_v1,ok,primary},{state,132734035421654268938420595158803047683595436032,riak_kv_multi_backend,false,{state,[{<<"leveldb_mult">>,riak_kv_eleveldb_backend,{state,<<>>,"/var/lib/riak/leveldb/132734035421654268938420595158803047683595436032",[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,35},{use_bloomfilter,true},{write_buffer_size,21073168}],[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{data_root,"/var/lib/riak/leveldb"},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{included_applications,[]},{
@qix
qix / gist:232389bd8b4012e6c553
Created July 28, 2015 00:25
oot@riak10:/home/josh# riak-admin cluster status
---- Cluster Status ----
Ring ready: true
+----------------------------------------+-------+-------+-----+-------+
| node |status | avail |ring |pending|
+----------------------------------------+-------+-------+-----+-------+
| (C) [email protected] |leaving| up | 21.5| 16.7 |
| [email protected] |leaving| up | 11.9| 16.7 |
| [email protected] |leaving| up | 16.8| 16.7 |
@qix
qix / gist:9f86b2b380266860f789
Created July 29, 2015 20:12
riak crash report
2015-07-29 19:12:51 =CRASH REPORT====
crasher:
initial call: riak_core_vnode:init/1
pid: <0.28158.0>
registered_name: []
exception exit: {{{badmatch,{reply,{riak_kv_w1c_put_reply_v1,ok,primary},{state,598016783243797190163421821199338462144370835456,riak_kv_multi_backend,false,{state,[{<<"leveldb_mult">>,riak_kv_eleveldb_backend,{state,<<>>,"/var/lib/riak/leveldb/598016783243797190163421821199338462144370835456",[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,35},{use_bloomfilter,true},{write_buffer_size,22216967}],[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{data_root,"/var/lib/riak/leveldb"},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false
@qix
qix / gist:9b8ddc70cc292103ca80
Created July 29, 2015 20:13
2015-07-29 20:11:00 =SUPERVISOR REPORT====
Supervisor: {local,riak_core_handoff_sender_sup}
Context: child_terminated
Reason: vnode_died
Offender: [{pid,<0.25521.9>},{name,riak_core_handoff_sender},{mfargs,{riak_core_handoff_sender,start_link,undefined}},{restart_type,temporary},{shutdown,brutal_kill},{child_type,worker}]
2015-07-29 20:11:00 =ERROR REPORT====
** Generic server <0.25378.9> terminating
** Last message in was {'$gen_cast',{work,{fold,#Fun<riak_kv_multi_backend.13.132002736>,#Fun<riak_kv_vnode.37.91169427>},{server,undefined,{<0.25521.9>,#Ref<0.0.25.251807>}},<0.25375.9>}}
** When Server state == {state,riak_kv_worker,{state,687933387884272662670093837274656655736483872768}}
@qix
qix / shlld.py
Created September 8, 2015 20:13
from __future__ import absolute_import
import cPickle
import zlib
from hyperloglog import SlidingHyperLogLog
def shlld_from_string(string, error_rate, window_count):
lpfm = cPickle.loads(zlib.decompress(string)) if string else None
return SlidingHyperLogLog(
@qix
qix / fabfile.py
Created October 18, 2015 17:31
from __future__ import absolute_import
import os
from fabric.api import (
local,
task,
)
from util import (
status,
@qix
qix / gist:c073957079c137f91da7
Created February 4, 2016 23:53
$ sudo pip3 install tooler
$ git clone [email protected]:qix/tooler && cd tooler
$ ./example/tool.py named:nginx* disk.free
nginx2$ df -mah | grep '/$'
nginx3$ df -mah | grep '/$'
nginx7$ df -mah | grep '/$'
nginx6$ df -mah | grep '/$'
nginx1$ df -mah | grep '/$'
nginx5$ df -mah | grep '/$'
nginx4$ df -mah | grep '/$'
@qix
qix / sada.md
Created May 6, 2016 22:41

We've built an example (simple) rocksdb server available at: http://authbox-cdn.s3.amazonaws.com/mirror/sada-sliding

The server should run inside an ubuntu docker container with flags:

./sada-sliding --port 9015 --rocksdb_db_path /data \
    --rocksdb_create_if_missing_one_off --version_timestamp_ms 9999999999999

This will start it up, and create the database on the first run. What we want is