1066145 notes

gistfile1.txt

Not clear where to send syslog data, and anyway we'd probably want to filter somewhat first.

Installed Heka directly from https://github.com/mozilla-services/heka/releases/download/v0.8.0/heka-0_8_0-linux-amd64.rpm on hp1

Heka can't do syslog-over-UDP at all, so it has to be fronted by something which dumps to disk

Overall plan:
 - get rsyslog to log something reasonable to disk - RSYSLOG_FileFormat seems close
   - maybe add fromhost?
   - maybe add severity, facility
 - configure heka to snarf that with the 'rsyslog' decoder
   - maybe add an extra filter to choose best of the two hostnames from rsyslog (fqdn > hostname > ip)
 - configure heka to output to rabbitmq using creds found in infra puppet
   - use a filter to add Fields['Random']

Filtering:
 - do most stuff in Heka
 - siphon CEF off to syslog hosts with rsyslog

rsyslog config

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

# The aggregation ruleset
# (note that this must be set up before it's referenced by $InputUDPServerBindRuleset)

$RuleSet aggregation

# the overall idea here is to divide the messages we receive between
# auditd-based CEF content (which goes to a syslog server) and the rest
# (which goes to MozDef via Heka)

if $programname == 'audit-cef' and $syslogfacility-text == 'local5' then @syslog1.private.scl3.mozilla.com
& ~

# everything else goes to an output file with rotation.  Heka will tail
# this file and inject the results into MozDef.
$outchannel aggregate,/opt/aggregator/log/to-heka.log,53687091200,/opt/aggregator/bin/rotate
# there appears to be no way to escape a tab, so these are literal tab charcters.  Copy-pasta
# beware!
$template heka,"%TIMESTAMP:::date-rfc3339%      %HOSTNAME%      %fromhost%      %programname%   %pri%   %syslogtag%     %syslogfacility-text%   %msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
*.* :omfile:$aggregate;heka

# set the ruleset back for subsequent rules
$RuleSet RSYSLOG_DefaultRuleset

# enable incoming UDP syslog, sending it to the 'aggregation' ruleset to
# avoid the default local processing
$ModLoad imudp.so
$InputUDPServerBindRuleset aggregation
$UDPServerRun 514