Developing with logstash

logstash.md

logstash v2.x

rake bootstrap
...
./bin/plugin install --no-verify logstash-codec-json
./bin/plugin install --no-verify logstash-output-stdout
./bin/plugin install --no-verify logstash-intput-log4j

sample config for geOrchestra sp

input {
  log4j {
    mode => server
    port => 4712
  }
}
filter {
   grok { match => [ "message", "%{COMMONAPACHELOG} (?<time.needed>(%{BASE10NUM}))" ] }
   mutate {
     convert => {
       "time.needed" => "float"
       "response"    => "integer"
       "bytes"       => "integer"
     }
   }
}

output {
  stdout {
    codec => json
  }
  elasticsearch {
    embedded => true
  }
}

conf sample (wmts.conf)

# input
input { stdin { } }

# filter
filter { 
   # First, waiting for varnish log file formats (combined apache logs)
   grok { match => [ "message", "%{COMBINEDAPACHELOG}" ] }
   # Then, parameters 
   grok {
     match => ["request", "(?<gis.version>([0-9\.]{5}))\/(?<gis.layer>([a-z0-9\.-]*))\/default\/(?<gis.release>([0-9]{8}))\/(?<gis.reference-system>([0-9]*))\/(?<gis.zoomlevel>([    0-9]*))\/(?<gis.row>([0-9]*))\/(?<gis.col>([0-9]*))\.(?<gis.filetype>([a-zA-Z]*))"]
   }
   wmts { }
}

# output
output {
  stdout { 
    debug => true 
    codec => json
  }
}
``

conf sample (wms.conf)

# input
input { stdin { } }

# filter
filter { 
   # First, waiting for varnish log file formats (combined apache logs)
   grok { match => [ "message", "%{COMBINEDAPACHELOG}" ] }

   # unlike the wmts plugin, the wms plugin does not use grok to parse
   # the request before calling the filter, because the parameters given
   # in the URI can be given in an arbitrary order.

   wms { }
}

# output
output {
  stdout { 
    debug => true 
    codec => json
  }
}

Testing WMTS filter

$ make clean
$ make flatjar
$ java -jar build/logstash-1.4.0.dev-modified-flatjar.jar agent -f wmts.conf --pluginpath ./lib/

Here are some samples of WMTS requests:

1.2.3.4 - - [23/Jan/2014:06:51:53 +0100] "GET http://wmts3.geo.admin.ch/1.0.0/ch.swisstopo.pixelkarte-farbe/default/20130903/21781/23/159/384.jpeg HTTP/1.1" 200 24657 "http://map.wanderland.ch/" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36"
1.2.3.4 - - [23/Jan/2014:06:51:53 +0100] "GET http://wmts0.geo.admin.ch/1.0.0/ch.swisstopo.pixelkarte-farbe/default/20130903/21781/23/161/384.jpeg HTTP/1.1" 200 37894 "http://map.wanderland.ch/" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36"
1.2.3.4 - - [23/Jan/2014:06:51:53 +0100] "GET http://wmts3.geo.admin.ch/1.0.0/ch.swisstopo.pixelkarte-farbe/default/20130903/21781/23/157/385.jpeg HTTP/1.1" 200 24936 "http://map.wanderland.ch/" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36"

Testing WMS filter:

$ java -jar build/logstash-1.4.0.dev-modified-flatjar.jar agent -f wms.conf --pluginpath ./lib/

Here are some queries to paste:

1.2.3.4 - - [23/Jan/2014:06:51:57 +0100] "GET http://tile2.schweizmobil.ch/mapproxy/service/?FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&SRS=EPSG:21781&BBOX=650400,222000,676000,247600&WIDTH=256&HEIGHT=256&LAYERS=WanderlandEtappenNational,WanderlandEtappenRegional,WanderlandEtappenLokal,WanderlandEtappenHandicap,Wanderwegnetz&z=17 HTTP/1.1" 200 12059 "http://map.schweizmobil.ch/" "Dalvik/1.6.0 (Linux; U; Android 4.2.1; ASUS Transformer Pad TF300T Build/JOP40D)"
1.2.3.4 - - [23/Jan/2014:06:52:21 +0100] "GET http://wms-mistra.bgdi.admin.ch/?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetCapabilities HTTP/1.1" 403 202 "http://www.esri.com/3B80AB13-36D3-4D6E-A602-39AB17542C6E" "ArcGIS Client Using WinInet"
1.2.3.4 - - [23/Jan/2014:06:52:02 +0100] "GET http://map.veloland.ch/wms?FORMAT=image%2Fpng&LAYERS=VelolandRoutenLokal_highlight,VelolandEtappenLokal_highlight,VelolandRoutenRegional_highlight,VelolandEtappenRegional_highlight,VelolandEtappenNational_highlight,VelolandRoutenNational_highlight&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&EXTRA_LINE=OFF&SRS=EPSG%3A21781&LAYER_VELOLANDROUTENNATIONAL=5&BBOX=662738,252898.5,666938,254901&WIDTH=1680&HEIGHT=801 HTTP/1.1" 200 5309 "http://map.veloland.ch/?lang=de&route=all" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"

How to launch a specific unit test:

java -jar build/logstash-1.4.0.dev-modified-flatjar.jar rspec spec/filters/wmts.rb