Splunk - Data input routing.md

Splunk - Data input routing

How to map logs received from multiple hosts on the same local port to the correct sourcetype and index.

Scenario

The Splunk data input UDP:514 receives events from the following devices:

• 10.0.0.1: A Netfilter firewall
• 10.0.0.2: A Squid proxy instance (1/2)
• 10.0.0.3: A Squid proxy instance (2/2)

We want to separate logs comming from the Firewall (10.0.0.1) and the Proxies (10.0.0.2 and 10.0.0.3):

• Firewall logs (10.0.0.1) in their own index (firewall) and sourcetype (netfilter)
• Proxies logs (10.0.0.2 and 10.0.0.3) in their common index (proxy) and sourcetype (squid)

[10.0.0.1] ---> UDP:514 --\                                                                      /--> index: firewall, sourcetype: netfilter
[10.0.0.2] ---> UDP:514 ---> [Splunk] ---> [inputs.conf] ---> [props.conf] ---> [transforms.conf]
[10.0.0.3] ---> UDP:514 --/                                                                      \--> index: proxy, sourcetype: squid

Data input - inputs.conf

We define a root input which receive logs on port UDP:514 and forward the events to the index syslog with the sourcetype syslog.

[udp://514]
index = syslog
sourcetype = syslog

Rules - props.conf

We define one stanza for each source or source group (using the source IP or hostname).

[host::10.0.0.1]
TRANSFORMS-netfilter = index_firewall, sourcetype_netfilter

[host::10.0.0.(2|3)]
TRANSFORMS-squid = index_proxy, sourcetype_squid

Transformations - transforms.conf

We define two stanzas per source: one for the index, one for the sourcetype. Notice the leading underscore _MetaData in the index_ stanzas, and not for the sourcetype_ stanzas.

[index_firewall]
REGEX = .*
DEST_KEY = _MetaData:Index
FORMAT = firewall

[sourcetype_netfilter]
REGEX = .*
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::netfilter

[index_proxy]
REGEX = .*
DEST_KEY = _MetaData:Index
FORMAT = proxy

[sourcetype_squid]
REGEX = .*
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::squid

Updated document using real-world sources (Squid and Netfilter).