Automate_ELK.MD

ELK Stack

First create an ELK stack for testing. I used the Bitnami one provided on GCP, which installs all three ELK (Elasticsearch, Logstash and Kibana) tools on to the same box, ssh onto the ELK box.

Create a Logstash conf file - as shown below:

input {
  http {
    host => "0.0.0.0" # default: 0.0.0.0
    port => 8080 # default: 8080
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "chef-automate-%{+dd.MM.YYYY}"
  }
  stdout { codec => rubydebug }
  file {
    path => "chef-automate-%{+dd.MM.YYYY}"
  }
}

I called this file http-input.conf

The input makes use of the Logstash http input plugin (https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html), which creates a web listener that Chef Automate can send data to, from its data tap.

I have not created a filter section, but you could if you only want to send a subset of the data in the output section.

The output is configured to send the data received from A2 to elasticsearch and also to stdout and a file for debugging purposes.

Start up Logstash with:

logstash -f http-input.conf

And wait for it to report that it is listening, takes 30 to 40 seconds.

ssh on to your Chef Automate box to configure the chef/data-feed-service:

cd /hab/pkgs/chef/data-feed-service/1.0.0/20200131184756/default.toml

Change the [service] section to speed up the reporting cadence and report all the nodes all the time (see highlighted).

[service]
host = "localhost"
port = 14001
feed_interval = "3m"
asset_page_size = 100
reports_page_size = 1000
node_batch_size = 100
updated_nodes_only = false

It doesn't appear that the config is exposed yet to ./chef-automate and changing the default.toml was the only way that I could find to make the config settings permanent.

Stop the service - chef/data-feed-service

hab svc chef/data-feed-service stop

Chef Automate will restart it.

./chef-automate status

Watch for it to come back up.

Create and test the Data Tap. Login to Chef Automate in your browser and type "beta" and then turn on the "ServiceNow CMDB Integration". Refresh your browser and you should see "Data Feeds" on the "Settings" page. Click "Create Data Feed" and give your feed destination a name and the fill in the IP address of your ELK server for xx.xx.xx.xx. Fill in Username and Password because you have to, they will not be used though.

Click "Send a test", if your Logstatch listener is set correctly and you have port 8080 open you should see two things:

In the browser:

Logstash debug output:

{
          "text" => "TEST: Successful validation completed by Automate",
      "@version" => "1",
    "@timestamp" => 2020-02-19T15:55:25.447Z,
       "headers" => {
        "accept_encoding" => "gzip",
           "request_path" => "/",
           "content_type" => "application/json",
              "http_host" => "xx.xx.xx.xx:8080",
         "request_method" => "POST",
         "content_length" => "60",
        "http_user_agent" => "Go-http-client/1.1",
            "http_accept" => "application/json",
           "http_version" => "HTTP/1.1"
    },
          "host" => "34.250.152.5"
}

Save the Data Feed by clicking "Save Destination".

Sometime in the next three minutes you will see the data tap send all your node run and compliance data (if you have any, if not time to create some!). That data will also be in Elasticsearch as well.

I had one issue where the maximum number of data indexes was exceeded, I was seeing nearly 1200 and the default max is 1000.

To list your elasticsearch indices, ssh onto the ELK box and issue:

curl http://localhost:9200/_cat/indices

Output:

chef-automate-19.02.2020 jTPbGNK7TD2CVaKTmjLruA 1 1 3 0 31.3kb 31.3kb

To increase the index size:

curl -X PUT http://localhost:9200/chef-automate-19.02.2020/_settings -d '{"index.mapping.total_fields.limit": 2000}' -H "Content-Type: application/json"

Design some visualisations and dashboards with the data that you have just put into Elasticsearch.

Open your browser to Kibana and click the Management Gear Wheel and then Index Patterns and finally Create Index Pattern. We want to index all the data for chef-automate-*

Click the Visualise icon, Create Visualisation and then create a pie, choose chef-automate-* as the source.

Add a "Split chart" bucket, set the Aggregation to Terms and search for "node_data.report.status.keyword" (this will only be here if you have compliance data). Repeat adding a split slice using the same data and run the pie and you will get something like this - I need lots of help to create Kibana visualisations!

tomrgio/Automate_ELK.MD