elasticsearch events parser

elastic.md

Intro

I will preface this with the fact that I am not a big data engineer and the methods described may or may not be best practice.

Install

I used a few different methods to run elk stack and what I found was https://github.com/deviantony/docker-elk

Follow steps in Readme for setup.

Setup Index.

In order to query an index you first need to create a mapping. The map below should cover most of the fields found in the events object.

1.) Go to dev tools http://localhost:5601/app/kibana#/dev_tools/console

2.) Paste the mapping below and push the play > arrow to run the transaction.

3.) After this command completes verify that the index was created. http://localhost:5601/app/kibana#/management/elasticsearch/index_management/indices

PUT /events
{
  "mappings": {
    "properties": {
      "apiVersion": {
        "type": "text"
      },
      "firstTimestamp": {
        "type": "date"
      },
      "count": {
        "type": "integer"
      },
      "kind": {
        "type": "text"
      },
      "lastTimestamp": {
        "type": "date"
      },
      "message": {
        "type": "text"
      },
      "reason": {
        "type": "text"
      },
      "reportingComponent": {
        "type": "text"
      },
      "reportingInstance": {
        "type": "text"
      },
      "source": {
        "properties": {
          "component": {
            "type": "text"
          },
          "host": {
            "type": "text"
          }
        }
      },
      "involvedObject": {
        "properties": {
          "kind": {
            "type": "text"
          },
          "name": {
            "type": "text"
          },
          "uuid": {
            "type": "text"
          }
        }
      },
      "metadata": {
        "properties": {
          "creationTimestamp": {
            "type": "date"
          },
          "name": {
            "type": "text"
          },
          "namespace": {
            "type": "text"
          },
          "resourceVersion": {
            "type": "text"
          },
          "selfLink": {
            "type": "text"
          },
          "uuid": {
            "type": "text"
          }
        }
      }
    }
  }
}

If that completes successful you should see a response like below.

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "events"
}

Add data

Next we want to add data to the index. Many differnt ways exist (logstash) I am adding the method I found to be the easiest for my usecase which was to download the events directly from a CI run.

example using events from cluster

1.) get events data from cluster. oc get events -o json &> events.json 2.) import events into elasticseatch index with the script below.

#!/bin/bash

# note index name {"_index": "events"} should match your index name greated above.
cat ./events.json \
  | jq -c '.items[] | {"index": {"_index": "events"}}, .'\
  | curl -H "Content-Type: application/json" \
  -XPOST localhost:9200/_bulk --data-binary @-

create index pattern

In order to query your index you ned to create an index pattern for your indexs.

1.) http://localhost:5601/app/kibana#/management/kibana/index_patterns click create Index pattern

2.) in the case where your index is called events you would want to name it something like event*

query data

1.) http://localhost:5601/app/kibana#/discover

NOTE Auth

depending on your setup you might have auth enabled. My example assumes you have auth disabled. You can pass the defaults set with docker-compose to curl by adding --user user:pw (elastic:changeme) thanks @retroflexer

• https://www.elastic.co/guide/en/elasticsearch/reference/7.1/http-clients.html