Skip to content

Instantly share code, notes, and snippets.

@excalq
Last active September 18, 2018 13:49
Show Gist options
  • Save excalq/0c5023fce8af90089040daa6404525f2 to your computer and use it in GitHub Desktop.
Save excalq/0c5023fce8af90089040daa6404525f2 to your computer and use it in GitHub Desktop.
A Logstash Grok Pattern for Rails 3
####################################
### Rais3 Log Grok (Unicorn and Thin supported at :info level)
## There are many non-capturing groups. For debugging, change them to named groups and use http://grokconstructor.appspot.com/do/match
#
## Assumes Rails is prepending :uuid to log lines, with one log per unicorn worker.
## Assumes the whole request is sent multiline (by Filebeat, not Logstash mulitline plugin)
#
## Add this to the Filebeat.yml prospector
## multiline:
# pattern: '\[[a-fA-F0-9]{32}\]( Started |$)'
## negate: true
## match: after
#
# To use this, save this gist to /etc/logstash/patterns.d/rails, and add this to your Logstash config:
# grok {
# patterns_dir => ["/etc/logstash/patterns.d/rails"]
# match => { "message" => "%{RAILS3}" }
# }
####################################
# Fragment patterns
RUUID \[\h{32}\]
RTIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE}
RAILSCONTROLLER (?<controller>[^#]+)#(?<action>\w+)
RAILSEXCLASS (?<exception>\w+((?:\:\:)?\w+)?)
RAILSPROFILE %{NUMBER:duration}ms(?: \((?:Views: %{NUMBER:view}ms \| )?ActiveRecord: %{NUMBER:db}ms\))?
# Whole-line/multiline patterns (faster to parse)
RPARAMETERS (?:^%{RUUID} Parameters: {%{DATA:params}}$\n)?
# In case rack_timer gem is used.
RAILSRACK (^(?:%{RUUID} )?(?:Rack Timer|source=rack-timeout).*?$\n)+
RAILSEXCEPTION (?:^%{RUUID}\s*\n%{RAILSEXCLASS} (- (?<error_message>.*)|\((?<message>.*)\))\:\n\s+(?<backtrace>(?m)%{GREEDYDATA})\n{2,3})
RAILSOTHER (?:^(?!(?:%{RUUID} )?Completed %{NUMBER}).*?$\n)
RAILSLOGCONTENT ^(?:(%{RAILSEXCEPTION}|%{RAILSRACK}|%{RAILSOTHER})*)
# Highest level patterns
RAILSHEAD ^%{RUUID} Started %{WORD:verb} "%{URIPATHPARAM:path}" for %{IPORHOST:remote_ip} at %{RTIMESTAMP:timestamp}$\n
RAILSPROCESSING ^%{RUUID} Processing by %{RAILSCONTROLLER} as %{WORD:format}$\n
RAILSFOOT ^%{RUUID} Completed %{NUMBER:status}%{DATA} in %{RAILSPROFILE}%{DATA}$\n
# Finally, use this as the master pattern which you'll grok in your Logstash conf
RAILS3 \A%{RAILSHEAD}%{RAILSPROCESSING}?%{RPARAMETERS}?%{RAILSLOGCONTENT}?%{RAILSFOOT}?%{RAILSRACK}?
@excalq
Copy link
Author

excalq commented Dec 24, 2016

Using .rb for syntax highlighting. Save this file as /etc/logstash/patterns.d/rails

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment