yaauie

Starting up

One pipeline has high worker_utilization, but hasn't been up for long enough for that to be considered a problem

> GET /_health_report HTTP/1.1
> Host: localhost:9600
> User-Agent: curl/8.7.1
> Accept: */*
> 

yaauie

Setup

First, I created a simplified pipeline using a generator input to produce exactly one event whose message is the value of an environment variable, and a cipher filter that similarly uses a key from an environment variable:

input {
  generator {
    count => 1
    message => "${CIPHERTEXT}"
  }

yaauie

Useful for finding field names from large schemas:

╭─{ rye@perhaps:~/src/elastic/scratch/ecs-20240517 }
╰─○ ./flatten_elasticsearch_legacy_template.rb < "${ecs_source?:}/generated/elasticsearch/legacy/template.json" | grep -e '\bip\b'
client.ip
client.nat.ip
destination.ip
destination.nat.ip
host.ip

yaauie

Use a single jvm options file with multiple versions of Java. Simply prefix a line with a specification of which versions it should be included with.

• X:: emitted when target version is exactly X
• X-:: emitted when target version is at least X
• -Y:: emitted when target version is not greater than Y
• X-Y: emitted when target version is at least X and not greater than Y

    8:option that only works with java 8

yaauie

package net.biesemeyer.streams;

import java.util.Set;
import java.util.function.BiConsumer;
import java.util.function.BinaryOperator;
import java.util.function.Function;
import java.util.function.Supplier;
import java.util.stream.Collector;
import java.util.stream.Stream;

yaauie

###############################################################################
# utf8-coerce.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to forcefully coerce string-value field
# to valid UTF-8, preferring a _representational_ transcode operation, and
# falling back to the use of UTF8 replacement characters when encountering byte
# sequences that cannot be represented in unicode, optionally stashing a base64
# encoded copy of the original when such lossy replacements are made.
###############################################################################
#

yaauie

Logstash Ruby Filter Scripts

• Transpose: transpose an array of two-value maps in a source field into an unordered key/value map, optionally storing the result in a target instead of overwriting (e.g., from [{"key":"this","value":"that"}] to {"this" => "that"})
• Untranspose: transpose an unordered key/value map in a source field into an array of two-value maps, optionally storing the result in a target instead of overwriting (e.g., from {"this" => "that"} to [{"key":"this","value":"that"}])
• Flatten Structure: transforms the deeply nested structure of an event or a source field into a flat key-value map by joining nested keys on a configurable separator, optionally storing the result in a target field

yaauie

# encoding: utf-8

require 'thread'
require 'monitor'

##
# The FairEnoughRouter is a generic "fair enough" router. When instantiated
# with a collection of objects, it can be used to select an arbitrary value,
# prioritising ones for which it hasn't recently been exceptional and
# those that are currently less-concurrently used.

yaauie

	\xA7	\xE9
ASCII-8BIT	(Encoding::UndefinedConversionError) "\xA7" from ASCII-8BIT to UTF-8	(Encoding::UndefinedConversionError) "\xE9" from ASCII-8BIT to UTF-8
Big5	(Encoding::InvalidByteSequenceError) incomplete "\xA7" on Big5	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on Big5
Big5-HKSCS	(Encoding::InvalidByteSequenceError) incomplete "\xA7" on Big5-HKSCS	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on Big5-HKSCS
Big5-UAO	(Encoding::InvalidByteSequenceError) incomplete "\xA7" on Big5-UAO	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on Big5-UAO
CESU-8	(Encoding::InvalidByteSequenceError) "\xA7" on CESU-8	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on CESU-8
CP51932	(Encoding::InvalidByteSequenceError) incomplete "\xA7" on CP51932	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on CP51932
CP850	º	Ú
CP852	ž	Ú

yaauie

Implicit AWS Credentials Test

The provided script is capable of invoking the AWS SDK directly from within the Logstash environment, and is useful for validating the SDK's ability to use implicit credentials (such shared credentials from a discoverable file on disk or credentials made available to an EC2 instance by IMDS or ECS).

It does so by writing an object to an S3 bucket, which the implicit credentials it finds needs to have write access to.

USAGE