Rye Biesemeyer yaauie

We have had some success using LS-to-LS over HTTP(S), which supports an HTTP(s) Load Balancer or Proxy in the middle, and can be secured with TLS/SSL. It can be made to be quite performant, but doing so requires some specific tuning.

Upstream (HTTP Output)

The upstream pipelie would contain a single HTTP output plugin aimed either directly at a downstream Logstash or at a Load Balancer, importantly configured with:

format => json_batch (for performance; without this one event will be sent at a time) and
retry_non_idempotent => true (for resilience; without this, some failures cannot be safely retried).

Depending on whether we ar sending directly to another Logstash or through an SSL-terminating Load Balancer or proxy, the output may need to be configured

with HTTP Basic credentials (user/password),

The included apply-template.rb provides a way generate Logstash config fragments from a shared template.

This can be useful for shared verbose configuration that is shared across multiple pipelines.

For example, if we are using multiple pipelines with pipelines.yml

 - pipeline.id: one
   path.config: "${LOGSTASH_HOME}/pipelines/one/*.conf"
 - pipeline.id: two

	###############################################################################
	# determine-field-type.logstash-filter-ruby.rb
	# ---------------------------------
	# A script for a Logstash Ruby Filter to determine a field's type
	###############################################################################
	#
	# Copyright 2022 Ry Biesemeyer
	#
	# Permission is hereby granted, free of charge, to any person obtaining a copy
	# of this software and associated documentation files (the "Software"), to deal

	literal: 👍
	u-encoded: \u001c\u{1F44D}
	inspected: "\u001C👍"
	dumped: "\x1C\u{1F44D}"
	json-encoced: "\u001c👍"

	#!/bin/sh
	#
	# Utility for determining why we cannot list the
	# contents of a deeply-nested directory.
	#
	# Usage:
	# listup.sh /deeply/nested/path
	#
	##############################################################################
	# Copyright 2022 Ry Biesemeyer

	###############################################################################
	# flatten-structure.logstash-filter-ruby.rb
	# ---------------------------------
	# A script for a Logstash Ruby Filter to flatten a nested structure to produce
	# flat structure whose keys are the paths of the previous structure
	###############################################################################
	#
	# Copyright 2022 Ry Biesemeyer
	#
	# Permission is hereby granted, free of charge, to any person obtaining a copy

	package org.logstash.instrument.metrics.timer;

	import org.logstash.instrument.metrics.AbstractMetric;

	import java.util.concurrent.TimeUnit;
	import java.util.concurrent.atomic.AtomicReference;
	import java.util.function.LongSupplier;
	import java.util.function.Supplier;

	/**

 ###############################################################################
 # replace-with-serialzied.logstash-filter-ruby.rb
 # ---------------------------------
 # A script for a Logstash Ruby Filter to replace the event's contents with a
 # single field containing a string JSON-encoded representation of the event.
 #
 # This filter _MUTATES_ the event, removing all DATA-keys while leaving METADATA
 # in-tact.
 #
 ###############################################################################

	╭─{ yaauie@limbo:~/src/elastic/ls (✔ 8.0) }
	╰─● (rbenv shell jruby-9.2.19.0; ruby kwargs-splat.rb)
	ArgumentError: wrong number of arguments calling `initialize` (given 1, expected 0)
	new at kwargs-splat.rb:7
	<main> at kwargs-splat.rb:17
	( rbenv shell jruby-9.2.19.0; ruby kwargs-splat.rb; ) 9.23s user 0.53s system 405% cpu 2.412 total
	[error: 1]

	###############################################################################
	# json-sanitize-field-names.logstash-filter-ruby.rb
	# ---------------------------------
	# A script for a Logstash Ruby Filter to transform a JSON string so that the
	# resulting JSON string's decoded representation does not contain square
	# brackets in keys.
	#
	# This filter does NOT parse the JSON string into an Object, and has undefined
	# behaviour when the string is not valid JSON.
	#