ES Magic

elasticsearch wizardry

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_most_important_queries_and_filters.html

Notable fields:

@fields.program <-- Will have the container.log, for instance infra2_nova_api_os_compute_container-1a69f4e5.nova-api-os-compute
@message <-- The actual log line
@timestamp <-- Timestamp
os_level <-- log level (INFO, WARN, etc)

Structure of single-query request:

{
    "_source" : [ "Field1", "Field2"],                  #Comma seperated list of _source fields to return, if left out returns all fields.
    "query" : {                                         #Start of Query line
        "match" : {                                     #Start match clause
            "@message" : "Thing you are grepping for"   #Define what we are matching
        }
    },
    "size": "Number of Results you want Returned",      #Defaults to 10 if left out
    "from": "Offset to start results from"              #Defaults to 0, starting at 10 will return results 10-X, etc.
}


Structure of multi-query request:

{
    "_source" : [ "Field1", "Field2"],                  #Comma seperated list of _source fields to return, if left out returns all fields.
    "query" : {                                         #Start of Query line
        "bool" : {                                      #Start of bool Query for multi-query requests
            "must" : [                                  #Start array of what must match. must_not and should are other options. Should is an "OR" clause.
                "match": {                              #Define what you are matching
                    "program" : {                       #Specify what we are matching
                        "query" : "Search1 Search2",    #You can specify multiple search words using another query, likely the thing we're looking for
                        "operator" : "and" }            #The default is OR. AND makes it require both
                    },                                  #Don't forget your Comma
                "match" : {
                    "@message" : "SEARCH_TERM"          #Your UUID or whatever you would normally grep a log for
                }]
        }
    },
    "size": "Number of Results you want Returned",      #Defaults to 10 if left out
    "from": "Offset to start results from"              #Defaults to 0, starting at 10 will return results 10-X, etc.
}




Example:

Note that the IP should be the VIP for the LB on the management network. 


**Find all Log entries for a single instance UUID**
http GET http://172.29.236.1:9200/_search <<< '{ "_source" : [ "@fields.program", "@message"], "query" : { "match" : {"@message" : "acd45117-d0b1-462d-a0b2-31d492de4945" }}}'

**Find nova api log entries across all days for an instance ID**
http GET http://172.29.236.1:9200/_search <<< '{ "_source": ["@fields.program", "@message" ], "query" : { "bool" : { "must": [{ "match" : { "program": { "query" : "nova api", "operator": "and" }}}, {"match": { "@message" : "acd45117-d0b1-462d-a0b2-31d492de4945" }} ] }}}'

**Find nova api log entries across a single day for an instance ID**
http GET http://172.29.236.1:9200/logstash-2015.01.28/_search <<< '{ "_source": ["@fields.program", "@message" ], "query" : { "bool" : { "must": [{ "match" : { "program": { "query" : "nova api", "operator": "and" }}}, {"match": { "@message" : "acd45117-d0b1-462d-a0b2-31d492de4945" }} ] }}}'

**Find all Log entries for a single instance within a range of time**
http GET http://172.29.236.1:9200/_search <<< '{ "_source": ["@fields.program", "@message" ], "query" : { "bool" : { "must": [ {"match": { "@message" : "acd45117-d0b1-462d-a0b2-31d492de4945" }}, {"range" : { "@timestamp" : { "gte" : "2015-01-28T15:30:00", "lte" : "2015-01-28T23:50:00" }}  } ] }}}'

**Find all nova api log entries for a single instance within a range of time**
http GET http://172.29.236.1:9200/_search <<< '{ "_source": ["@fields.program", "@message" ], "query" : { "bool" : { "must": [ { "match" : { "program": { "query" : "nova api", "operator": "and" }}},  {"match": { "@message" : "acd45117-d0b1-462d-a0b2-31d492de4945" }}, {"range" : { "@timestamp" : { "gte" : "2015-01-28T15:30:00", "lte" : "2015-01-28T23:50:00" }}  } ] }}}'