Splunk Tips and Tricks

Splunk.md

Splunk

Helpful tips and tricks for Splunk.

Formatting

Splunk uses the | ("or bar") as a means to break up statements. Instead of using one long string of statements, consider deliminating | [statement] on seperate lines.

Example

index=rh_jboss host=gss-diag*.web.prod*
| transaction host startswith="Starting processing of documentation message..." endswith="interrupted due to"
| rex field=_raw ".+Started processing documentation with id \[(?<doc>[^\]]+)\]"
| rex field=_raw ".+in current environment \[(?<locale>[^\]]+)\]"
| rex field=_raw ".+Trying to (?<action>[^\[]+)\[(?<url>[^\]]+)\]"
| rex field=_raw ".+Received \[(?<http_status>[^\]]+)\].+with message \[(?<failure>[^\]]+)\]"
| rex field=_raw ".+Message processing of \[(?<msg>[^\]]+)\]"
| table doc, locale, url, http_status, failure, action, msg

Cron

Splunk cron settings are just like *nix cron settings fields:

1. Minute: 0-59
2. Hour: 0-23
3. Day of the month: 1-31
4. Month: 1-12
5. Day of the week: 0-6 (where 0 = Sunday)

Transactions

When performing transactions, it may be desirable to consume regular expressions from each line within the transaction. The documentation doesn't readily explain how to do this. However it turns out to be very simple:

[index=some index] [host=some host]
| transaction [field] startsWith="some start string" endsWith="some end string"
| rex field=_raw "your reg ex for a line (?<val1>...)"
| rex field=_raw "your reg ex for another line (?<val2>...)"
| rex field=_raw "your reg ex for yet another line (?<val3>...)"
| table val1, val2, val3

Example

index=rh_jboss host=gss-diag*.web.prod*
| transaction host startswith="Starting processing of documentation message..." endswith="interrupted due to"
| rex field=_raw ".+Started processing documentation with id \[(?<doc>[^\]]+)\]"
| rex field=_raw ".+in current environment \[(?<locale>[^\]]+)\]"
| rex field=_raw ".+Trying to (?<action>[^\[]+)\[(?<url>[^\]]+)\]"
| rex field=_raw ".+Received \[(?<http_status>[^\]]+)\].+with message \[(?<failure>[^\]]+)\]"
| rex field=_raw ".+Message processing of \[(?<msg>[^\]]+)\]"
| table doc, locale, url, http_status, failure, action, msg

Negative Look Aheads

Negative look aheads is useful when your reg ex's fail with the following type of error:

[splunk-host] Streamed search execute failed because: Error in 'rex' command: regex="Some Reg Ex" has exceeded configured match_limit, consider raising the value in limits.conf.

Use something akin to: (?!Something that should be excluded)

Example

index=rh_jboss host=gss-diag*prod* Pyxis "Message processing of"
| rex field=_raw "Message processing of \[(?!interrupted due to)(?<message>.+)\].+interrupted due to \[(?<reject>.+)(\])?"
| dedup message
| table _time, reject, message