hackathon quick hack to get into any Elastic

quick_and_dirty_zeek_into_elastic.md

Understanding of Zeek Fields

use the following OSSEM branch

Prep Elasticsearch

You only need to do this one time skip this section if you have done this once already and go to the "Upload data section"

Login/browse to your Kibana instance Go to Dev Tools (which is the wrench icon in the bottom left)

Copy and paste the following items and enter them. To enter it, either press the button that looks like a play button near top right of the input. or press Ctrl + Enter

PUT /_template/temporary_hackathon
{
  "order": 11,
  "index_patterns": [ "indexme-zeek-hackathon" ],
  "version": 2020050201,
  "settings": {
    "index": {
      "mapping": {
        "ignore_malformed": true,
        "total_fields.limit": "5000",
        "coerce": true
      }
    },
    "refresh_interval": "5s",
    "number_of_replicas": 0,
    "number_of_shards": 1
  },
  "mappings": {
    "dynamic": "true",
    "dynamic_templates": [
      {
        "strings": {
          "match_mapping_type": "string",
          "mapping": {
            "ignore_above": 12048,
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          }
        }
      }
    ],
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "id.orig_h": {
        "type": "ip"
      },
      "id.resp_h": {
        "type": "ip"
      }
    }
  }
}

PUT _ingest/pipeline/temporary_hackathon
{
  "description" : "convert epoch to timestmap",
  "processors" : [
    {
      "date" : {
        "field" : "ts",
        "target_field" : "@timestamp",
        "formats" : ["ISO8601"],
        "if": "ctx.containsKey('ts') && ctx.containsKey('_write_ts')"
      }
    },
    {
      "date" : {
        "field" : "ts",
        "target_field" : "@timestamp",
        "formats" : ["UNIX"],
        "if": "ctx.containsKey('ts') && !(ctx.containsKey('@timestamp'))"
      }
    }
  ]
}

Upload Data

Get the zeek logs (clone repo or individual download from github interface)

Open the CybherChef Recipe that will allow you to extract and setup the zeek logs: zeek logs zipped to elasticsearch upload

Drag and drop zeek files (you do not need to unzip them) into the site that opens up.

Copy the output and paste it into Kibana Dev tools and enter it.

if you get an error such as Failed to execute 'setItem' on "storage': Setting the value of .......... then enable the very last Tail recipe at the end which will give you 1000-2000, just readd POST indexme-zeek-hackathon/_bulk?pipeline=temporary_hackathon before you copy and paste into dev tools again. then change the value to 2000 and so on

Understanding of Zeek Fields

use the following OSSEM branch

Other ways to explore the data

json to csv https://json-csv.com/ save as excel and go

brim: https://github.com/brimsec/brim

wireshark, portable version is nice if you don't want to od a full admin install: https://www.wireshark.org/download.html

just found this recently, haven't looked into it yet: https://github.com/SuperCowPowers/zat

Filebeat example

replace paths below, with the location of your zeek logs

###################### Filebeat Zeek/Corelight Configuration Example #########################
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-reference-yml.html
#----------------------------- Input Logs --------------------------------
filebeat.inputs:
- type: log
  enabled: true
  # Change this to the directory of where your Zeek logs are stored
  paths:
    - /usr/share/zeek/logs/*.log
  #json.keys_under_root: true
  #fields_under_root: true
#----------------------------- Kafka output --------------------------------
output.kafka:
  # Place your HELK IP(s) here (keep the port).
  hosts: ["<HELK-IP>:9092"]
  topic: "zeek"
  max_message_bytes: 1000000

Quick and dirst setup of ELK

download elasticsearch:

https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.2-linux-x86_64.tar.gz
tar -zxvf elasticsearch-7.6.2-linux-x86_64.tar.gz

cd elasticsearch-7.6.2
./bin/elasticsearch

^ keep above terminal running

open new terminal

download kibana: https://artifacts.elastic.co/downloads/kibana/kibana-7.6.2-linux-x86_64.tar.gz

tar -zxvf kibana-7.6.2-linux-x86_64.tar.gz
cd kibana-7.6.2-linux-x86_64/
./bin/kibana

Browse to http://localhost:5601