yaauie

Implicit AWS Credentials Test

The provided script is capable of invoking the AWS SDK directly from within the Logstash environment, and is useful for validating the SDK's ability to use implicit credentials (such shared credentials from a discoverable file on disk or credentials made available to an EC2 instance by IMDS or ECS).

It does so by writing an object to an S3 bucket, which the implicit credentials it finds needs to have write access to.

USAGE

yaauie

###############################################################################
# determine-field-type.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to determine a field's type
###############################################################################
#
# Copyright 2022 Ry Biesemeyer
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal

yaauie

literal: 👍
u-encoded: \u001c\u{1F44D}
inspected: "\u001C👍"
dumped: "\x1C\u{1F44D}"
json-encoced: "\u001c👍"

yaauie

#!/bin/sh
#
# Utility for determining why we cannot list the
# contents of a deeply-nested directory.
#
# Usage:
#  listup.sh /deeply/nested/path
#
##############################################################################
# Copyright 2022 Ry Biesemeyer

yaauie

###############################################################################
# flatten-structure.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to flatten a nested structure to produce
# flat structure whose keys are the paths of the previous structure
###############################################################################
#
# Copyright 2022 Ry Biesemeyer
#
# Permission is hereby granted, free of charge, to any person obtaining a copy

yaauie

We have had some success using LS-to-LS over HTTP(S), which supports an HTTP(s) Load Balancer or Proxy in the middle, and can be secured with TLS/SSL. It can be made to be quite performant, but doing so requires some specific tuning.

Upstream (HTTP Output)

The upstream pipelie would contain a single HTTP output plugin aimed either directly at a downstream Logstash or at a Load Balancer, importantly configured with:

• format => json_batch (for performance; without this one event will be sent at a time) and
• retry_non_idempotent => true (for resilience; without this, some failures cannot be safely retried).

Depending on whether we ar sending directly to another Logstash or through an SSL-terminating Load Balancer or proxy, the output may need to be configured

• with HTTP Basic credentials (user/password),

yaauie

package org.logstash.instrument.metrics.timer;

import org.logstash.instrument.metrics.AbstractMetric;

import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.LongSupplier;
import java.util.function.Supplier;

/**

yaauie

###############################################################################
# replace-with-serialzied.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to replace the event's contents with a
# single field containing a string JSON-encoded representation of the event.
#
# This filter _MUTATES_ the event, removing all DATA-keys while leaving METADATA
# in-tact.
#
###############################################################################

yaauie

The included apply-template.rb provides a way generate Logstash config fragments from a shared template.

This can be useful for shared verbose configuration that is shared across multiple pipelines.

For example, if we are using multiple pipelines with pipelines.yml

 - pipeline.id: one
   path.config: "${LOGSTASH_HOME}/pipelines/one/*.conf"
 - pipeline.id: two

yaauie

╭─{ yaauie@limbo:~/src/elastic/ls (✔ 8.0) }
╰─● (rbenv shell jruby-9.2.19.0; ruby kwargs-splat.rb)
ArgumentError: wrong number of arguments calling `initialize` (given 1, expected 0)
     new at kwargs-splat.rb:7
  <main> at kwargs-splat.rb:17
( rbenv shell jruby-9.2.19.0; ruby kwargs-splat.rb; )  9.23s user 0.53s system 405% cpu 2.412 total
[error: 1]