Logstash Elasticsearch Template

logstash-template.json

{
    "template": "logstash-*",
    "settings" : {
        "number_of_shards" : 1,
        "number_of_replicas" : 0,
        "index" : {
            "query" : { "default_field" : "@message" },
            "store" : { "compress" : { "stored" : true, "tv": true } }
        }
    },
    "mappings": {
        "_default_": { 
            "_all": { "enabled": false },
            "_source": { "compress": true },
            "dynamic_templates": [
                {
                    "string_template" : { 
                        "match" : "*",
                        "mapping": { "type": "string", "index": "not_analyzed" },
                        "match_mapping_type" : "string"
                     } 
                 }
             ],
             "properties" : {
                "@fields": { "type": "object", "dynamic": true, "path": "full" }, 
                "@message" : { "type" : "string", "index" : "analyzed" },
                "@source" : { "type" : "string", "index" : "not_analyzed" },
                "@source_host" : { "type" : "string", "index" : "not_analyzed" },
                "@source_path" : { "type" : "string", "index" : "not_analyzed" },
                "@tags": { "type": "string", "index" : "not_analyzed" }, 
                "@timestamp" : { "type" : "date", "index" : "not_analyzed" },
                "@type" : { "type" : "string", "index" : "not_analyzed" }
            }
        }
    }
}

set-template.sh

curl -XPUT 'http://localhost:9200/_template/template_logstash/' -d @logstash-template.json

Is this a pretty generic 'I was lazy and just used embedded but now I need it to be a bit quicker' config?

@thattommyhall: I'll be using a slight variation of this template in production soon. I only adjusted the number of shards and replicas. Going from 1 shard to 2 shards cost me little in size but spreads out the data between my 2 ES nodes, and replicas: 1 means I can have redundancy)

Hi,
Could you please help converting this json into a yml format.
I wish to include these settings into elasticsearch.yml

Thanks

@sunny2385 create a directory called templates/ underneath your config directory and put the JSON file there. It will be auto-loaded next time Elasticsearch starts.

Note: there appears to be a bug whereby templates loaded from config are not visible (under some conditions) when using curl - see this post

Could you explain "store" : { "compress" : { "stored" : true, "tv": true } } a little bit please, I could not found corresponding documentation

@eclipse9614 It's pretty basic, but the settings are mentioned here: http://www.elasticsearch.org/guide/reference/index-modules/store/

Do you use Kibana to query your ES indexes?

I've tried your template against ES 0.90.5 and Kibana 3.0.0milestone4, and any indexes created using the template don't return any data to Kibana. The data is loaded into the index, and I run basic queries against it successfully myself, but if I try the query Kibana runs then no data is returned.

I'll carry on investigating myself, but was interested to know if you've seen the same problem.

For anybody who is interested - the line that was preventing Kibana from querying the index is:
"query" : { "default_field" : "@message" },

Remove that, re-index and Kibana works fine.

FYI i added:
"numeric_detection" : true,
to get automatically numeric fields (integer, long, float...).

@gregmruphy
how re-index? Please tell me the way

thanks !