Skip to content

Instantly share code, notes, and snippets.

@Zohorul
Forked from sfloess/Splunk.md
Created April 14, 2020 18:41
Show Gist options
  • Save Zohorul/41ddd4613ea60e3f8dc10ec4d4d419df to your computer and use it in GitHub Desktop.
Save Zohorul/41ddd4613ea60e3f8dc10ec4d4d419df to your computer and use it in GitHub Desktop.
Splunk Tips and Tricks

Splunk

Helpful tips and tricks for Splunk.

Formatting

Splunk uses the | ("or bar") as a means to break up statements. Instead of using one long string of statements, consider deliminating | [statement] on seperate lines.

Example

index=rh_jboss host=gss-diag*.web.prod*
| transaction host startswith="Starting processing of documentation message..." endswith="interrupted due to"
| rex field=_raw ".+Started processing documentation with id \[(?<doc>[^\]]+)\]"
| rex field=_raw ".+in current environment \[(?<locale>[^\]]+)\]"
| rex field=_raw ".+Trying to (?<action>[^\[]+)\[(?<url>[^\]]+)\]"
| rex field=_raw ".+Received \[(?<http_status>[^\]]+)\].+with message \[(?<failure>[^\]]+)\]"
| rex field=_raw ".+Message processing of \[(?<msg>[^\]]+)\]"
| table doc, locale, url, http_status, failure, action, msg

Cron

Splunk cron settings are just like *nix cron settings fields:

  1. Minute: 0-59
  2. Hour: 0-23
  3. Day of the month: 1-31
  4. Month: 1-12
  5. Day of the week: 0-6 (where 0 = Sunday)

Transactions

When performing transactions, it may be desirable to consume regular expressions from each line within the transaction. The documentation doesn't readily explain how to do this. However it turns out to be very simple:

[index=some index] [host=some host]
| transaction [field] startsWith="some start string" endsWith="some end string"
| rex field=_raw "your reg ex for a line (?<val1>...)"
| rex field=_raw "your reg ex for another line (?<val2>...)"
| rex field=_raw "your reg ex for yet another line (?<val3>...)"
| table val1, val2, val3

Example

index=rh_jboss host=gss-diag*.web.prod*
| transaction host startswith="Starting processing of documentation message..." endswith="interrupted due to"
| rex field=_raw ".+Started processing documentation with id \[(?<doc>[^\]]+)\]"
| rex field=_raw ".+in current environment \[(?<locale>[^\]]+)\]"
| rex field=_raw ".+Trying to (?<action>[^\[]+)\[(?<url>[^\]]+)\]"
| rex field=_raw ".+Received \[(?<http_status>[^\]]+)\].+with message \[(?<failure>[^\]]+)\]"
| rex field=_raw ".+Message processing of \[(?<msg>[^\]]+)\]"
| table doc, locale, url, http_status, failure, action, msg

Negative Look Aheads

Negative look aheads is useful when your reg ex's fail with the following type of error:

[splunk-host] Streamed search execute failed because: Error in 'rex' command: regex="Some Reg Ex" has exceeded configured match_limit, consider raising the value in limits.conf.

Use something akin to: (?!Something that should be excluded)

Example

index=rh_jboss host=gss-diag*prod* Pyxis "Message processing of"
| rex field=_raw "Message processing of \[(?!interrupted due to)(?<message>.+)\].+interrupted due to \[(?<reject>.+)(\])?"
| dedup message
| table _time, reject, message

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment