Splunk Notes

splunk.md

SPLUNK & Logging

Tips

• SET TIMEZONE IN PROFILE

creating a basic saved search

sourcetype=syslog (host=east OR host=europe OR host=asia OR host=jobs) marklar
| regex _raw="^.* (east|asia|europe|jobs) [\-a-zA-Z]*marklar[^:]*:"

using the search in the web ui

| savedsearch marklar

refining the search

| savedsearch marklar
| rex field=_raw "^.* (east|asia|europe|jobs) (?<app>[\-a-zA-Z]*marklar[^:]*): (?<rest>.*)$"
| rex field=rest "consultationParticipantId='(?<cpid>[^']+)'"
| rex field=rest "pinNumber='(?<pin>[^']+)'"

dial out times

marklar: checkForClient='true' | rex field=_raw "differenceInMinutes='(?<diff>[^']+)'" | stats count by diff

dialOutTimes (filtered)

marklar: | rex field=_raw "differenceInMinutes='(?<diff>[0-9]+)'" | search diff <= 30 | stats count by diff

find log events count for all marklar instances by env, app

| savedsearch marklar-fields
| stats count by env, app

(bar / split) visualize the split by of app in each env

| savedsearch marklar-fields
| stats count(eval(env="east")) as east
  , count(eval(env="europe")) as europe
  , count(eval(env="asia")) as asia
  , count(eval(env="jobs")) as job
  by app

roundtrip comparison

| savedsearch marklar
| search roundtripInSeconds='*'
| timechart count as Bottle
| appendcols
  [search dialOutToCouncilMember: checkForClient='true'
  | timechart count as Legacy]

epiquery templates called

| savedsearch marklar-fields
| rex field=rest "template: '(?<template>[^']+)'"

figure out log size in MB of apps on starphleet

(host=east OR host=asia OR host=europe) earliest=-6h latest=now
| rex field=_raw " (east|west|asia|europe) (?<app>[^:]+):"
| search app!="CRON*"
| eval raw_len=len(_raw)
| eval rounded_len=raw_len / 1024 / 1024
| stats count, sum(rounded_len) AS MB by app
| sort - MB

last week

earliest=-1d@d latest=-0d@d marklar*: dialOutToCouncilMember:ok
| eval ReportKey="today"
| append [search earliest=-8d@d latest=-7d@d
  marklar*: dialOutToCouncilMember:ok
  | eval ReportKey="yesterday"
  | eval new_time=_time+86400*7]
| eval _time=if(isnotnull(new_time), new_time, _time)
| timechart span=1h count by ReportKey

web logs

| savedsearch marklar | search /diagnostic

not so helpful logging examples

marklar*: undefined.json
express:
TypeError:
undefined
GET /
host=east "GET /*" (NOT "GET /diag*")

http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Abstract http://docs.splunk.com/Documentation/Splunk/6.2.5/Search/Specifytimemodifiersinyoursearch http://dev.splunk.com/view/logging-best-practices/SP-CAAADP6