andrewkroh

Breaking change guide to field data types

Fleet packages specify mappings that are used to create index component templates. When a Fleet package is updated the component template is replaced, and a new backing index is created by rolling over the data stream. Field mapping changes are implemented this way because it is generally not possible to change the field mappings of an existing index after data has been indexed. So this results in a single data stream that is composed to backing indices that have different data types for the same field.

When a query is executed on the data stream it may span multiple backing indices, therefore we should avoid having mixed data types for a field that result in incompatibilities at query-time.

Examples of ways queries can break due to conflicting types

• Term query where the value is not a valid IP, but one of the backing indices uses type: ip, (mixed ip and keyword types)
• Term query where a boolean field where the value is not a valid boolean (true, fal

andrewkroh

// Beats TLS configuration options.
package tls

$version: "v8.14.0"

#base64String: =~"^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==)?$"

#hexSHA256: =~"^[a-fA-F0-9]{64}$"

#pemCerts: =~"^(?:(?:-+BEGIN CERTIFICATE-+\\s+)(?:([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==)?\\s+)+(?:-+END CERTIFICATE-+\\s*))+$"

andrewkroh

---

filebeat.inputs:
  # Consume output from
  #  evtx_dump --dont-show-record-number -o xml <file.evtx> > /tmp/samples/file.evtx.xml
  # See https://github.com/omerbenamram/evtx.
  - type: filestream
    id: evtx_dump_xml
    parsers:
      - multiline:

andrewkroh

filebeat.inputs:
  - host: localhost:9514
    id: udp-extrahop-cef-9514
    type: udp
    processors:
      - convert:
          mode: copy
          fields:
            - { from: "message", to: "event.original" }

andrewkroh

---

filebeat.inputs:
  - type: cel
    id: config-123-watcher
    interval: 1m
    resource:
      url: file:///etc/conf.d/foo.conf
    program: |
      file(state.url).as(content, content.sha256().hex().as(hash, {

andrewkroh

filebeat.inputs:
  - type: journald
    processors:
      # For https://kubernetes.io/docs/concepts/cluster-administration/system-logs/#json-log-format
      - if:
          and:
            - equals.journald.process.name: kubelet
            - regexp.message: '^{'
        then:
          # 'kubelet' should be mapped as a flattened field in ES because

andrewkroh

processors:
  - script:
      # This uses a Beat script processor to include only ipv4 addresses
      # in the host.ip field. This would need to placed after the add_host_metadata
      # processor.
      #
      # It would be a lot more efficient to have add_host_metadata allow controlling
      # what addresses were included because this has to execute for every event.
      #
      # References:

andrewkroh

package main

import (
	"flag"
	"log"
	"os/user"
	"syscall"
	"unsafe"

	"golang.org/x/sys/windows"

andrewkroh

winlogbeat.event_logs:
  - name: Security
    ignore_older: 1h
    processors:
      - script:
          lang: javascript
          source: |
            var console = require("console");

            var ids = {

andrewkroh

DRAFT: Routing Filebeat data to a Fleet integration data stream

This is an unofficial tutorial that may be useful to users that are in the process of migrating to to Elastic Agent and Fleet. It explains the steps to route some Filebeat data into a data stream managed by a Fleet integration package.

Install the Fleet integration

Installing a Fleet integration sets up all of its data streams and dashboards. There are two methods to install. In these examples we install the Hashicorp Vault 1.3.1 integration.

Use Kibana (easiest)