Mark Walkom markwalkom

Elasticsearch

Sum number of docs in a cluster

cat nodes_stats.json|jq '.nodes[].indices.docs.count'|awk '{s+=$0} END {print s}'

Sum total store size

cat nodes_stats.json|jq '.nodes[].indices.store.size_in_bytes'|awk '{s+=$0} END {print s}'

Working with the swapi data

Get a list of planets + key for translate lookup

cat people.json | jq -r '.[]|"\"\(.pk)\"" + ": " + "\"\(.fields.name)\""'

This is an example of using ELK to parse and view collectd data.

Caveat - I haven't fully tested this mapping yet, it doesn't take into account any other fields that may be added with other collectd plugins, just the ones I have specified below.

ES 1.2.x settings

Name	Description
action.allow_id_generation	-
action.auto_create_index	-
action.bulk.compress	-
action.destructive_requires_name	http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/_parameters.html#_parameters
action.disable_shutdown	http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-shutdown.html#_disable_shutdown
action.get.realtime	http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-get.html#realtime

Keybase proof

I hereby claim:

I am markwalkom on github.
I am markwalkom (https://keybase.io/markwalkom) on keybase.
I have a public key whose fingerprint is 3624 D73D 1018 8785 6475 F84D 5CA5 78CB 1845 5C92

To claim this, I am signing this object:

	$ logstash-2.2.0/bin/plugin list
	logstash-codec-avro
	logstash-codec-cef
	logstash-codec-cloudfront
	logstash-codec-cloudtrail
	logstash-codec-collectd
	logstash-codec-compress_spooler
	logstash-codec-dots
	logstash-codec-edn
	logstash-codec-edn_lines

	F2B_DATE %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[ ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})
	F2B_ACTION (\w+)\.(?:\w+)(\s+)?\:
	F2B_JAIL \[(?<jail>\w+\-?\w+?)\]
	F2B_LEVEL (?<level>\w+)\s+

	input {
	elasticsearch {
	hosts => [ "HOSTNAME_HERE" ]
	port => "9200"
	index => "INDEXNAME_HERE"
	size => 500
	scroll => "5m"
	}
	}
	output {

	input {
	elasticsearch {
	hosts => [ "HOSTNAME_HERE" ]
	port => "9200"
	index => "INDEXNAME_HERE"
	size => 1000
	scroll => "5m"
	docinfo => true
	scan => true
	}

	# Via https://smelloworld.wordpress.com/2016/05/17/missing-fields-search-in-elasticsearch/

	{“query”:{“filtered”:{“query”:{“match_all”:{}},”filter”:{“missing”:{“field”:”FIELDNAME”}}}}}