Skip to content

Instantly share code, notes, and snippets.

@imweijh
imweijh / README.md
Created October 9, 2022 08:50 — forked from jordansissel/README.md
logstash internal messaging and queues

Logstash internals (Queues and Threading)

The logstash agent is 3 parts: inputs -> filters -> outputs.

Each '->' is an internal messaging system. It is implemented with a 'SizedQueue' in Ruby. SizedQueue allows a bounded maximum of items in the queue such that any writes to the queue will block if the queue is full at maximum capacity.

Logstash sets the queue size to 20. This means only 20 events can be pending into the next phase - this helps reduce any data loss and in general avoids logstash trying to act as a data storage system. These internal queues are not for storing messages long-term.

In reverse, here's what happens with a queue fills.

@imweijh
imweijh / check_mk_agent-ubuntu-install.sh
Created September 24, 2022 13:15 — forked from nitrag/check_mk_agent-ubuntu-install.sh
Setup check_mk_agent via systemd on Ubuntu 16.04 (OMD, Check_MK)
# How to install Check MK Agent on ubuntu 16.04
# Install check_mk_agent:
# - sudo apt-get install check-mk-agent (will install older version)
# - On your Check_MK dashboard, go to "Monitoring Agents", click the link for "Check_MK Agent for Linux", save the raw text
# on your server:
sudo vi /usr/bin/check_mk_agent
# paste Check_MK dashboard > Monitoring Agents > Check_MK Agent for Linux

Electron is tricky to get set up on Windows Subsystem for Linux, but it can work!

Four things needed overall:

  1. you need WSL2, not WSL1
  2. you need node, of course, and that part isn't so bad
  3. you need to apt install several dependencies
  4. you need an X Server so it can display the electron GUI over in Windows-land

Setup instructions, in order:

@imweijh
imweijh / logstashgsub.conf
Created December 17, 2021 09:59
logstash gsub regex replace use capture group
input {
generator {
lines => [
"whatever|1|BA|110100|11:19:39|1|101.00000|whatever|123456789||10.000|||61033|",
"whatever|1|BP|110100|11:19:39|1|101.00000|whatever|123456789||10.000|||61033|",
"whatever|1|SA|110100|11:19:39|1|101.00000|whatever|123456789||10.000|||61033|",
"whatever|1|SP|110100|11:19:39|1|101.00000|whatever|123456789||10.000|||61033|",
"whatever|1|BC|110100|11:19:39|1|101.00000|whatever|123456789||10.000|||61033|",
"whatever|1|SC|110100|11:19:39|1|101.00000|whatever|123456789||10.000|||61033|",
"whatever|1|BF|110100|11:19:39|1|101.00000|whatever|123456789||10.000|||",
@imweijh
imweijh / log4jclean.sh
Last active December 11, 2021 05:58
log4j clean jndi class
find / -iname 'log4j-core*.jar' -exec jar tf {} \; | grep org/apache/logging/log4j/core/lookup/JndiLookup.class
find / -iname 'log4j-core*.jar' -print0 | xargs -0 -I{} zip -d {} org/apache/logging/log4j/core/lookup/JndiLookup.class
find / -iname 'log4j-core*.jar' -exec jar tf {} \; | grep org/apache/logging/log4j/core/lookup/JndiLookup.class
@imweijh
imweijh / CNLohr's Guide for Windows C Apps in 2021.md
Created August 16, 2021 09:29 — forked from cnlohr/CNLohr's Guide for Windows C Apps in 2021.md
How to Set Up a Windows Computer to Write C applications in 2021

Building C apps on Windows in 2021

This document was written on April 3, 2021. The procedure may change over time. This is a companion gist to the youtube video here, where I go through every step of both options

Youtube Version Of This Document

@imweijh
imweijh / install_check_mk_agent.bat
Created April 23, 2021 03:44 — forked from Ham5ter/install_check_mk_agent.bat
This Script install the check-mk-agent and adds a Firewall Rule to the local Windows Firewall to allow Access to it!
::
:: This Script install the check-mk-agent and adds a Firewall Rule to the local Windows Firewall to allow Access to it!
:: https://gist.github.com/Ham5ter/2a8526e843c72ff9343bc3a38cdac97e
::
:: Allow ICMP Echo (Ping)
netsh advfirewall firewall delete rule name="ICMP Allow incoming V4 echo request"
netsh advfirewall firewall add rule name="ICMP Allow incoming V4 echo request" protocol=icmpv4:8,any dir=in action=allow
:: Allow TCP Port 6556 (check-mk-agent)
@imweijh
imweijh / logstash_filtersr _performance.md
Created April 22, 2021 01:27 — forked from colinsurprenant/logstash_filtersr _performance.md
measuring logstash filters performance

Measuring logstash filters performance

  • given a data file sample named data.log
  • modify your config to use the stdin input so that you can pipe you sample log file to logstash
input {
  stdin {
    codec => line
  }
}
@imweijh
imweijh / logstash_jdbc_oracle.conf
Last active May 7, 2021 02:54
logstash jdbc syn oracle table to kafka every second
input {
jdbc {
jdbc_driver_library => "D:\logstash-7.12.0\ojdbc8.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:@//127.0.0.1:1521/servicename"
jdbc_user => "user"
jdbc_password => "password"
tracking_column => "record_sn"
use_column_value => true
tracking_column_type => "numeric"
@imweijh
imweijh / logstash-grok-mlutiline-as-one-record.conf
Created March 22, 2021 08:17 — forked from madumalt/logstash-grok-mlutiline-as-one-record.conf
Process multiples lines of unstructured logs as one structured record by Logstash prior pushing them to Elasticsearch.
input {
beats {
port => 5301
}
}
filter {
if [fields][type] == "monthly-indexed.r2web.bsd.alljob" {
mutate {
gsub => ["message","/"," "]