logstash grok filter benchmark script

README.md

Requirements:

• JDK 8/9/11 (oracle or openjdk)

Steps to setup the scripts:

mkdir benches
cd benches
curl https://artifacts.elastic.co/downloads/logstash/logstash-7.6.0.tar.gz | tar -zxf - 
cd logstash-7.6.0
wget https://gist.github.com/jsvd/a2613ea1ba00f02926a302781ca62f7b/raw/1a9e97f7c20bf21fefe5b461238074c86859d993/benchmark_grok.rb
sed -i.bak 's/gem \"benchmark-ips\", :group => :development/gem \"benchmark-ips\"/g' Gemfile   
bin/logstash-plugin update

To run the benchmarks:

bin/ruby benchmark_grok.rb

benchmark_grok.rb

# encoding: utf-8

require_relative "lib/bootstrap/environment"
LogStash::Bundler.setup!({:without => [:build, :development]})
require "logstash-core"
require "logstash/environment"
require "logstash/plugin"
require 'benchmark/ips'

line = '220.181.108.96 - - [13/Jun/2015:21:14:28 +0000] "GET /blog/geekery/xvfb-firefox.html HTTP/1.1" 200 10975 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"'
pattern = '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'

# grok plugin setup
grok_plugin = LogStash::Plugin.lookup("filter", "grok").new("match" => ["message", pattern])
grok_plugin.register

# do the benchmark
Benchmark.ips do |x|

  x.time = 30
  x.warmup = 10

  x.report("grok_plugin") do
    # we need to create a new event on each iteration since it's modified by grok
    event = LogStash::Event.new("message" => line)
    grok_plugin.filter(event)
  end
  
  # x.report("grok_plugin_2") do
  #   event = LogStash::Event.new("message" => line)
  #   grok_plugin_2.filter(event)
  # end

  x.compare! # necessary only with multiple reports
end

Is this available as a docker image?

Thanks. For me I needed sudo for bundler: sudo gem install bundler

@esg-llachance, fwif I'm running it interactively in docker with this:
docker run -it --rm --name grok -v "$PWD":/benches -w /benches jruby /bin/bash

@brianjolly When I run rake bootstrap, it says bash: rake: command not found.

Running jruby -S gem install rake && jruby -S rake shows another error message, Gem::MissingSpecVersionError: Could not find 'json' (~> 1) - did find: [json-2.2.0-java]. Can I update the Gemfile somehow, or use another docker image?

Hi @phihag, these are my setup steps inside the container:

docker run -it --rm --name grok -v "$PWD":/benches -w /benches jruby /bin/bash
cd logstash
rake bootstrap
cd ..
bundle update --bundler
bundle install

bundle exec ruby benchmark_grok.rb

The script and instructions have been updated to run with logstash 7.6.0