Tim Conrad timconradinc

Cluster graphing

It's frequently useful to graph a metric over a cluster of hosts, e.g. "show me the number of requests/s being handled by all of my load balancers".
Doing this in vanilla Graphite is easy - it honors both glob expressions (lb*) and brace expressions ({a,b,c,d}).
But how do we generate these for clusters whose hostnames don't glob well, and/or whose members change over time?

Generally flippin' awesome.
Has a database of Metrics, Graphs composing 1+ Metrics, and Dashboards composing 1+ Graphs.

	ElasticSearch Quick Guide

	ElasticSearch for Logstash Overview

	There are two ways to send data to ElasticSearch from Logstash. The first is the 'elasticsearch' output and the other is
	the 'elasticsearch_http' output. In a nutshell, the 'elasticsearch' output is tightly coupled with your elasticsearch
	cluster, and the 'elasticsearch_http' output isn't.

	What does this mean? The 'elasticsearch' output will always start up a local ElasticSearch node and try to join it to
	your ElasticSearch cluster. This has the end goal of making Logstash aware of your cluster - if a node goes down,

	# this configuration is to parse the syslog files and then put them into redis.

	input {
	file {
	# this input is a dump of sendmail and proofpoint logs, it's really messy
	type => "proofpoint_raw_type"
	tags => "proofpoint_raw"
	path => ["/home/conrad/testlog.log"]
	}
	#stdin {

	# Force all memory to be locked, forcing the JVM to never swap
	bootstrap.mlockall: true

	# Indexing Settings for Writes
	indices.memory.index_buffer_size: 50%
	index.refresh_interval: 30
	index.translog.flush_threshold_ops: 50000
	index.store.compress.stored: true
	index.routing.allocation.total_shards_per_node: 6

	ES Connectivity -
	- Should respond properly when can't write to ES
	- Should tell user if they can't connect to ES

	Events
	- Fields should be able to slide wider/narrower & 'close' the panel when not needed.

	Editing
	- Saving should default to same name, or easily add name of current config
	- 'Create Panel' shows up below the scroll. Perhaps a better place would be opposite of 'create panel' -

	[beaver]
	redis_url:redis://10.99.9.99:6379/0

	[/syslog/logs/palo/user-infolog]
	type: PaloAltoTrafficLog

	[/syslog/logs/palo/locallog]
	type: PaloAltoThreatLog

	# this is for testing/troubleshooting

	<!-- the env variables are controlled by Chef and passed in via -D on the java command-line -->
	<!-- This is using the appender here: https://github.com/t0xa/gelfj -->
	<appender name="graylog2" class="org.graylog2.log.GelfAppender">
	<param name="graylogHost" value="${graylog.server}"/>
	<param name="originHost" value="${graylog.origin}"/>
	<param name="extractStacktrace" value="true"/>
	<param name="addExtendedInformation" value="true"/>
	<!-- The _web part is because a given app has multiple components -->
	<!-- This app might have a _web as well as an _batch component -->
	<param name="facility" value="${graylog.facility}_web"/>

	[beaver]
	redis_url:redis://10.5.88.6:6379/0