Grok Apache2 Custom-Pattern (streamlined Custom-Log to comply Error Log and enhanced Custom Log by adding more information)

gistfile1.txt

filter {
  grok {
    pattern => ['(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGHOST:logsource}) (?:%{SYSLOGPROG}): (?<messagebody>\[%{DAY} %{MONTH} %{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{YEAR}\] \[%{LOGLEVEL:severity}\] \[client %{IP:a2_client_ip}\] (?:Request: \"%{GREEDYDATA:a2_request}\" %{INT:a2_http_code} (?:%{GREEDYDATA:a2_sent_bytes}|%{INT:a2_sent_bytes}) %{INT:a2_response_time} %{GREEDYDATA:a2_referer}\" \"%{GREEDYDATA:a2_user_agent}\" \"%{GREEDYDATA:a2_ssl_protocol}\" \"%{GREEDYDATA:a2_ssl_cipher}\"|Request: \"%{GREEDYDATA:a2_request}\" %{INT:a2_http_code} (?:%{GREEDYDATA:a2_sent_bytes}|%{INT:a2_sent_bytes}) %{INT:a2_response_time} \"%{GREEDYDATA:a2_referer}\" \"%{GREEDYDATA:a2_user_agent}\")|%{GREEDYDATA})']
  type => "apache2"
  }
}

# Apache 2 Logformat for customlog with SSL/TLS Logging (last part "rsysloghostnames" can be changed to anything else)
LogFormat "[%{%a %b %d %H:%M:%S %Y}t] [info] [client %h] Request: \"%m http(s)://%{Host}i%U%q %H\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\" \"%{SSL_PROTOCOL}x\" \"%{SSL_CIPHER}x\"" rsysloghostnamesssl

# Example central Syslog Entry for Pattern
2013-04-09T11:33:35+02:00 apachehost9 a2_ruby_vhost1: [Tue Apr 09 11:33:35 2013] [info] [client 10.10.200.12] Request: "GET http(s)://ruby-app.example.com/subdir/?id=1 HTTP/1.1" 200 2524 808 "http://ruby-app.example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:20.0) Gecko/20100101 Firefox/20.0" "TLSv1" "RC4-SHA"

Updated to also log SSL Protocol and used SSL Cipher

Further enhanced Grok Pattern to support more types of log messages and now separates the messagebody from the header

in Revision 8 i streamlined the loglevel from a2_loglevel to the central severity field (easier to search!)