imweijh

Latency Comparison Numbers (~2012)
----------------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD

imweijh

input {
	beats {
		port => 5301
	}
}

filter {
	if [fields][type] == "monthly-indexed.r2web.bsd.alljob" {
		mutate {
            gsub => ["message","/"," "]

imweijh

Measuring logstash filters performance

• given a data file sample named data.log
• modify your config to use the stdin input so that you can pipe you sample log file to logstash

input {
  stdin {
    codec => line
  }
}

imweijh

::
:: This Script install the check-mk-agent and adds a Firewall Rule to the local Windows Firewall to allow Access to it!
:: https://gist.github.com/Ham5ter/2a8526e843c72ff9343bc3a38cdac97e
::

:: Allow ICMP Echo (Ping)
netsh advfirewall firewall delete rule name="ICMP Allow incoming V4 echo request"
netsh advfirewall firewall add rule name="ICMP Allow incoming V4 echo request" protocol=icmpv4:8,any dir=in action=allow

:: Allow TCP Port 6556 (check-mk-agent)

imweijh

Building C apps on Windows in 2021

This document was written on April 3, 2021. The procedure may change over time. This is a companion gist to the youtube video here, where I go through every step of both options

imweijh

Electron is tricky to get set up on Windows Subsystem for Linux, but it can work!

Four things needed overall:

1. you need WSL2, not WSL1
2. you need node, of course, and that part isn't so bad
3. you need to apt install several dependencies
4. you need an X Server so it can display the electron GUI over in Windows-land

Setup instructions, in order:

imweijh

# How to install Check MK Agent on ubuntu 16.04

# Install check_mk_agent:
#   - sudo apt-get install check-mk-agent (will install older version)
#   - On your Check_MK dashboard, go to "Monitoring Agents", click the link for "Check_MK Agent for Linux", save the raw text
#     on your server:

sudo vi /usr/bin/check_mk_agent
# paste Check_MK dashboard > Monitoring Agents > Check_MK Agent for Linux

imweijh

Logstash internals (Queues and Threading)

The logstash agent is 3 parts: inputs -> filters -> outputs.

Each '->' is an internal messaging system. It is implemented with a 'SizedQueue' in Ruby. SizedQueue allows a bounded maximum of items in the queue such that any writes to the queue will block if the queue is full at maximum capacity.

Logstash sets the queue size to 20. This means only 20 events can be pending into the next phase - this helps reduce any data loss and in general avoids logstash trying to act as a data storage system. These internal queues are not for storing messages long-term.

In reverse, here's what happens with a queue fills.

imweijh

We have had some success using LS-to-LS over HTTP(S), which supports an HTTP(s) Load Balancer or Proxy in the middle, and can be secured with TLS/SSL. It can be made to be quite performant, but doing so requires some specific tuning.

Upstream (HTTP Output)

The upstream pipelie would contain a single HTTP output plugin aimed either directly at a downstream Logstash or at a Load Balancer, importantly configured with:

• format => json_batch (for performance; without this one event will be sent at a time) and
• retry_non_idempotent => true (for resilience; without this, some failures cannot be safely retried).

Depending on whether we ar sending directly to another Logstash or through an SSL-terminating Load Balancer or proxy, the output may need to be configured

• with HTTP Basic credentials (user/password),

imweijh

http {
  log_format bodylog '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" $request_time '
      '<"$request_body" >"$resp_body"';

  lua_need_request_body on;

  set $resp_body "";
  body_filter_by_lua '