Pete Fritchman fetep

logstash config snippet:

output {
    elasticsearch {
        host => "localhost"
        node_name => "logstash-test1"
    }
}

# This tests the raw speed of logstash passing messages around between
# threads/queues without doing much logic.

input {
    generator {
        type => "gen"
        count => 1000000
    }
}

make statsd better for high throughput

In the interest of shipping high volumes of updates in the shortest amount of time and work, I'm proposing two statsd changes:

enhance the timer update format
add a zeromq input (in addition to UDP)

Original UDP protocol

over here, https://gist.github.com/3243702, i proposed extending the text protocol by supporting updating a timer with multiple values at once (to help statsd clients that do a bit of batching for efficiency). this makes the update string a bit expensive to parse (especially because you have to get to the end of a potentially long string to even know what kind of update you have).

so here are some thoughts on a different way to encode the statsd update.

if we chose one of these, so we could call this protocol version "2" in our zeromq input (e.g. zmq message payload would be "2;").

Keybase proof

I hereby claim:

I am fetep on github.
I am fetep (https://keybase.io/fetep) on keybase.
I have a public key whose fingerprint is ABBC E079 C8DC 883A 5697 32DC 8EBD 9865 6367 8B88

To claim this, I am signing this object:

	#!/usr/bin/env ruby

	require 'benchmark'

	N=1000 # number of parse iterations
	T=10000 # number of timer update samples

	# build some sample timer update strings w/#{T} values
	samples = []
	T.times { samples << ((Kernel.rand(1000) + 100.0) / 100.0) }

	{
	"template": "logstash-*",
	"order" : 0,
	"settings" : {
	"number_of_shards" : 6,
	"number_of_replicas" : 1,
	"index" : {
	"routing" : { "allocation" : { "total_shards_per_node" : "2" } },
	"query" : { "default_field" : "@message" },
	"store" : { "compress" : { "stored" : true, "tv" : true, "throttle" : { "type" : "merge", "max_bytes_per_sec" : "5mb" } } },

	c0 fetep-vm(~) % cat bin/git-push-n
	#!/bin/bash

	if [ $# -ne 2 ]; then
	echo "usage: git push-n <remote> <ref>" >&2
	exit 1
	fi

	output=$(git push -n "$@" 2>&1)
	git_rc=$?

	0 fetep-x201(~/l/logstash) % cat etc/bench.conf
	input {
	generator {
	type => "gen"
	count => 100000
	}
	}

	filter {
	noop {