SumoLogic Integration via Fluentd

fluent-sumo.md

Sumo - Fluentd Integration

There are a couple ways to integrate SumoLogic with fluentd.

In both cases, in SumoLogic configure an HTTP collector to send the log data to. For the log messages to be grouped properly, make sure to not enable multiline processing or enable one message per request. This configuration will allow the fluentd plugin and the lamdba function mentioned below to send multiple JSON messages at a time (bulk update), with SumoLogic separating each JSON message into distinct SumoLogic messages for proper aggregation, etc.

One thing to note is I am assuming all the log messages are structured as JSON objects - haven't tested this out with unstructred text messages.

Fluentd SumoLogic Plugin

Use of the sumlogic plugin is extremely straightforward - you basically need to know your collector endpoint and write it into your config. Note the http collector endpoint URLs need to be kept secret since anyone could log to that endpoint. You need to extract the host and path components from the URL obtained via SumoLogic and update your fluentd config accordingly.

Example:

<source>
  type tail
  path /home/vagrant/goxavi/src/github.com/xtracdev/xavisample/xs.log
  pos_file ./x-log-pos
  tag xavi
  time_key fake
  format json
</source>


<match **>
  @type copy

  <store>
  @type stdout
  </store>
  
  <store>
  @type sumologic
  host endpoint1.collection.us2.sumologic.com
  port 443
  format json
  path /receiver/v1/http/xxxxxx
  </source>
</match>

Note in the above stdout is used for debugging purposes - not needed for prodution. Also useful for debugging is running fluent with tracing enabled, e.g. fluentd -vv -c fluentd.conf

SumoLogic via Kinesis and Lambda

This one is more involved, but might be an option if you want to do other real time processing in Amazon on the log data. In this option you use the fluentd kinesis plugin to send to Amazon, then use a lamba function like k2sl_lambda to sent to Sumo.

In this case the fluentd config looks like

<source>
  type tail
  path /home/vagrant/goxavi/src/github.com/xtracdev/xavisample/xs.log
  pos_file ./x-log-pos
  tag xavi
  time_key fake
  format json
</source>


<match **>
  @type copy

  <store>
  @type kinesis
  stream_name LogStream
  aws_key_id xxxxxx
  aws_sec_key xxxxxx
  region us-east-1
  random_partition_key true
  </store>

  <store>
  @type stdout
  </store>
</match>

Just wanted to say thanks for this. I was a little confused about how to store logs after using @type stdout.