yaauie

Usage

Checkpoints:

Single Checkpoint:

lsq-cpdump "${LOGSTASH_HOME}/data/queue/main/checkpoint.head"

All Checkpoints for a given queue:

yaauie

Starting up

One pipeline has high worker_utilization, but hasn't been up for long enough for that to be considered a problem

> GET /_health_report HTTP/1.1
> Host: localhost:9600
> User-Agent: curl/8.7.1
> Accept: */*
> 

yaauie

Setup

First, I created a simplified pipeline using a generator input to produce exactly one event whose message is the value of an environment variable, and a cipher filter that similarly uses a key from an environment variable:

input {
  generator {
    count => 1
    message => "${CIPHERTEXT}"
  }

yaauie

Useful for finding field names from large schemas:

╭─{ rye@perhaps:~/src/elastic/scratch/ecs-20240517 }
╰─○ ./flatten_elasticsearch_legacy_template.rb < "${ecs_source?:}/generated/elasticsearch/legacy/template.json" | grep -e '\bip\b'
client.ip
client.nat.ip
destination.ip
destination.nat.ip
host.ip

yaauie

Use a single jvm options file with multiple versions of Java. Simply prefix a line with a specification of which versions it should be included with.

• X:: emitted when target version is exactly X
• X-:: emitted when target version is at least X
• -Y:: emitted when target version is not greater than Y
• X-Y: emitted when target version is at least X and not greater than Y

    8:option that only works with java 8

yaauie

package net.biesemeyer.streams;

import java.util.Set;
import java.util.function.BiConsumer;
import java.util.function.BinaryOperator;
import java.util.function.Function;
import java.util.function.Supplier;
import java.util.stream.Collector;
import java.util.stream.Stream;

yaauie

###############################################################################
# utf8-coerce.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to forcefully coerce string-value field
# to valid UTF-8, preferring a _representational_ transcode operation, and
# falling back to the use of UTF8 replacement characters when encountering byte
# sequences that cannot be represented in unicode, optionally stashing a base64
# encoded copy of the original when such lossy replacements are made.
###############################################################################
#

yaauie

Logstash Ruby Filter Scripts

• Transpose: transpose an array of two-value maps in a source field into an unordered key/value map, optionally storing the result in a target instead of overwriting (e.g., from [{"key":"this","value":"that"}] to {"this" => "that"})
• Untranspose: transpose an unordered key/value map in a source field into an array of two-value maps, optionally storing the result in a target instead of overwriting (e.g., from {"this" => "that"} to [{"key":"this","value":"that"}])
• Flatten Structure: transforms the deeply nested structure of an event or a source field into a flat key-value map by joining nested keys on a configurable separator, optionally storing the result in a target field

yaauie

# encoding: utf-8

require 'thread'
require 'monitor'

##
# The FairEnoughRouter is a generic "fair enough" router. When instantiated
# with a collection of objects, it can be used to select an arbitrary value,
# prioritising ones for which it hasn't recently been exceptional and
# those that are currently less-concurrently used.

yaauie

	\xA7	\xE9
ASCII-8BIT	(Encoding::UndefinedConversionError) "\xA7" from ASCII-8BIT to UTF-8	(Encoding::UndefinedConversionError) "\xE9" from ASCII-8BIT to UTF-8
Big5	(Encoding::InvalidByteSequenceError) incomplete "\xA7" on Big5	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on Big5
Big5-HKSCS	(Encoding::InvalidByteSequenceError) incomplete "\xA7" on Big5-HKSCS	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on Big5-HKSCS
Big5-UAO	(Encoding::InvalidByteSequenceError) incomplete "\xA7" on Big5-UAO	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on Big5-UAO
CESU-8	(Encoding::InvalidByteSequenceError) "\xA7" on CESU-8	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on CESU-8
CP51932	(Encoding::InvalidByteSequenceError) incomplete "\xA7" on CP51932	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on CP51932
CP850	º	Ú
CP852	ž	Ú