Last active
September 18, 2018 13:49
-
-
Save excalq/0c5023fce8af90089040daa6404525f2 to your computer and use it in GitHub Desktop.
A Logstash Grok Pattern for Rails 3
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#################################### | |
### Rais3 Log Grok (Unicorn and Thin supported at :info level) | |
## There are many non-capturing groups. For debugging, change them to named groups and use http://grokconstructor.appspot.com/do/match | |
# | |
## Assumes Rails is prepending :uuid to log lines, with one log per unicorn worker. | |
## Assumes the whole request is sent multiline (by Filebeat, not Logstash mulitline plugin) | |
# | |
## Add this to the Filebeat.yml prospector | |
## multiline: | |
# pattern: '\[[a-fA-F0-9]{32}\]( Started |$)' | |
## negate: true | |
## match: after | |
# | |
# To use this, save this gist to /etc/logstash/patterns.d/rails, and add this to your Logstash config: | |
# grok { | |
# patterns_dir => ["/etc/logstash/patterns.d/rails"] | |
# match => { "message" => "%{RAILS3}" } | |
# } | |
#################################### | |
# Fragment patterns | |
RUUID \[\h{32}\] | |
RTIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE} | |
RAILSCONTROLLER (?<controller>[^#]+)#(?<action>\w+) | |
RAILSEXCLASS (?<exception>\w+((?:\:\:)?\w+)?) | |
RAILSPROFILE %{NUMBER:duration}ms(?: \((?:Views: %{NUMBER:view}ms \| )?ActiveRecord: %{NUMBER:db}ms\))? | |
# Whole-line/multiline patterns (faster to parse) | |
RPARAMETERS (?:^%{RUUID} Parameters: {%{DATA:params}}$\n)? | |
# In case rack_timer gem is used. | |
RAILSRACK (^(?:%{RUUID} )?(?:Rack Timer|source=rack-timeout).*?$\n)+ | |
RAILSEXCEPTION (?:^%{RUUID}\s*\n%{RAILSEXCLASS} (- (?<error_message>.*)|\((?<message>.*)\))\:\n\s+(?<backtrace>(?m)%{GREEDYDATA})\n{2,3}) | |
RAILSOTHER (?:^(?!(?:%{RUUID} )?Completed %{NUMBER}).*?$\n) | |
RAILSLOGCONTENT ^(?:(%{RAILSEXCEPTION}|%{RAILSRACK}|%{RAILSOTHER})*) | |
# Highest level patterns | |
RAILSHEAD ^%{RUUID} Started %{WORD:verb} "%{URIPATHPARAM:path}" for %{IPORHOST:remote_ip} at %{RTIMESTAMP:timestamp}$\n | |
RAILSPROCESSING ^%{RUUID} Processing by %{RAILSCONTROLLER} as %{WORD:format}$\n | |
RAILSFOOT ^%{RUUID} Completed %{NUMBER:status}%{DATA} in %{RAILSPROFILE}%{DATA}$\n | |
# Finally, use this as the master pattern which you'll grok in your Logstash conf | |
RAILS3 \A%{RAILSHEAD}%{RAILSPROCESSING}?%{RPARAMETERS}?%{RAILSLOGCONTENT}?%{RAILSFOOT}?%{RAILSRACK}? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Using
.rb
for syntax highlighting. Save this file as /etc/logstash/patterns.d/rails