Grok ESXi 5.x Pattern (for Logstash) (including puppet format with special escaping!)

gistfile1.txt

filter {
 grok {
  pattern => ['(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGHOST:logsource}) (?:%{SYSLOGPROG}): (?<messagebody>(?:\[(?<esxi_thread_id>[0-9A-Z]{8,8}) %{DATA:esxi_loglevel} \'%{DATA:esxi_service}\'\] %{GREEDYDATA:esxi_message}|%{GREEDYDATA}))']
  type => "esxi"
 }
}

# Puppet format with escaping

pattern => [ "(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:.* (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}|(?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}): (?:(?:\[[0-9A-Z]{8,8}) (?:%{GREEDYDATA:esxi_loglevel}) \\\'(?:%{GREEDYDATA:esxi_service})\\\'] (?:%{GREEDYDATA:message})|(?:%{GREEDYDATA:message}))" ],

Hi Guys,

I have a problem with this not matching all messages from my vmware logs:

The below matches:

Jan 15 18:38:57 HOSTNAME Hostd: [6BBC3B90 info 'DvsTracker'] FetchUplinkDVPortgroups: added 24 items

The below does not match:

Jan 15 18:40:18 HOSTNAME Vpxa: [FFE3EB90 verbose 'VpxaHalCnxHostagent' opID=WFU-ac4ab664] [WaitForUpdatesDone] Received callback

Can you spot the problem ?

Sure. 😄

Ok, let's not keep the suspense too long.

The difference why the Hostd: output matches and Vpxa does not is the difference in the content between the first square brackets.

You can try something like this:

(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) %{SYSLOGHOST:esxi_hostname} %{SYSLOGPROG:esxi_program}(\[%{INT:esxi_pid}\])?: (?<messagebody>(?:\[(?<esxi_thread_id>[0-9A-Z]{8,8}) %{DATA:esxi_loglevel} \'%{DATA:esxi_service}\'(\s.*)?\] %{GREEDYDATA:esxi_message}|%{GREEDYDATA}))

This ESXi logging is quite a mess indeed. Every service seems to have an entirely different format.
That's why this attempt at capturing the output contains the "OR %GREEDYDATA" at the end in case the quite detailed filter starting at "messagebody" does not match, which it will not for Vpxa.

In any case I believe you should get acquainted with grok (http://www.logstash.net/docs/1.4.2/filters/grok) and also regular expressions.