imweijh

Logstash internals (Queues and Threading)

The logstash agent is 3 parts: inputs -> filters -> outputs.

Each '->' is an internal messaging system. It is implemented with a 'SizedQueue' in Ruby. SizedQueue allows a bounded maximum of items in the queue such that any writes to the queue will block if the queue is full at maximum capacity.

Logstash sets the queue size to 20. This means only 20 events can be pending into the next phase - this helps reduce any data loss and in general avoids logstash trying to act as a data storage system. These internal queues are not for storing messages long-term.

In reverse, here's what happens with a queue fills.

imweijh

# How to install Check MK Agent on ubuntu 16.04

# Install check_mk_agent:
#   - sudo apt-get install check-mk-agent (will install older version)
#   - On your Check_MK dashboard, go to "Monitoring Agents", click the link for "Check_MK Agent for Linux", save the raw text
#     on your server:

sudo vi /usr/bin/check_mk_agent
# paste Check_MK dashboard > Monitoring Agents > Check_MK Agent for Linux

imweijh

Electron is tricky to get set up on Windows Subsystem for Linux, but it can work!

Four things needed overall:

1. you need WSL2, not WSL1
2. you need node, of course, and that part isn't so bad
3. you need to apt install several dependencies
4. you need an X Server so it can display the electron GUI over in Windows-land

Setup instructions, in order:

imweijh

Building C apps on Windows in 2021

This document was written on April 3, 2021. The procedure may change over time. This is a companion gist to the youtube video here, where I go through every step of both options

imweijh

::
:: This Script install the check-mk-agent and adds a Firewall Rule to the local Windows Firewall to allow Access to it!
:: https://gist.github.com/Ham5ter/2a8526e843c72ff9343bc3a38cdac97e
::

:: Allow ICMP Echo (Ping)
netsh advfirewall firewall delete rule name="ICMP Allow incoming V4 echo request"
netsh advfirewall firewall add rule name="ICMP Allow incoming V4 echo request" protocol=icmpv4:8,any dir=in action=allow

:: Allow TCP Port 6556 (check-mk-agent)

imweijh

Measuring logstash filters performance

• given a data file sample named data.log
• modify your config to use the stdin input so that you can pipe you sample log file to logstash

input {
  stdin {
    codec => line
  }
}

imweijh

input {
	beats {
		port => 5301
	}
}

filter {
	if [fields][type] == "monthly-indexed.r2web.bsd.alljob" {
		mutate {
            gsub => ["message","/"," "]

imweijh

Latency Comparison Numbers (~2012)
----------------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD

imweijh

# this is an alternate compact function implementation which also removes keys with empty string values
# writing the tests for this is left as an excercise to the reader :D

def compact(h)
  h.inject({}) do |result, (k, v)|
    if v.is_a?(Hash)
      result[k] = compact(v)
    elsif v.is_a?(String)
      result[k] = v unless v.empty?
    elsif !v.nil?

imweijh

Set up geospatial scientific Python with Miniconda on Windows

Anaconda is an excellent, simple way to get Python up and running on your computer. But, it includes a lot of packages you'll never use but consume gigs and gigs of hard drive space. Instead, you can just install miniconda and then choose the individual packages you need. The steps below explain how to do this to set up a Python environment for geospatial data science. These steps are Windows-specific, but the same process works on Mac or Linux (just don't download the wheels from Gohlke - conda/pip install them directly). If you're having trouble, here are more detailed instructions on getting geopandas and geospatial Python up and running.

Install Miniconda

• Download the Miniconda for Python 3.5 installer
• Install miniconda to C:\Anaconda and set it as the system's default Python