README.md

Log aggregator sidecar pattern with Fluentd and Sumo Logic

This gist shows how to use a sidecar logging container to collect applicattion logs and ship them to Sumo Logic.

For this example we are using a Sumo Logic Hosted Collector and an HTTP Endpoint Source.

One note about this approach: In order to send the application logs to Sumo Logic we are using Fluentd as a sidecar container to collect and ship the logs. This is probably not the ideal solution since the ideal solution would, maybe, be a cluster-wide configuration to integrate with Sumo Logic. Although, with the config presented here, we keep all the necessary changes and related work at the application level.

Fluentd Sumo Logic plugin Docker image

Since we don't have a public image with the Fluentd Output Sumo Logic plugin we are following the instructions in the Fluentd Docker hub documentation to customize and install plugins.

FROM fluent/fluentd:v1.7-1

USER root

RUN apk add --no-cache --update --virtual .build-deps \
        sudo build-base ruby-dev \
 && sudo gem install fluent-plugin-sumologic_output \
 && sudo gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

USER fluent

Sidecar container with a logging agent

We are following the approach described in this documentation with the following adaptation.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluent.conf: |
    <source>
      @type tail
      format none
      path /var/log/application.log
      pos_file /var/log/application.log.pos
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      tag application-name.applogs
      read_from_head false
    </source>
    <match **.applogs>
      @type sumologic
      endpoint  "#{ENV['SUMO_ENDPOINT']}"
      log_format json
      source_category "#{ENV['SUMO_SOURCE_CATEGORY']}"
      source_name "#{ENV['SUMO_SOURCE_NAME']}"
      open_timeout 10
    </match>
---
apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  containers:
  - name: app-container
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date)" >> /var/log/application.log;
        i=$((i+1));
        sleep 1;
      done
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: fluentd-agent
    image: soeirosantos/fluentd-sumologic_output-plugin
    env:
    - name: SUMO_ENDPOINT
      value: "your-sumo-http-endpoint"
    - name: SUMO_SOURCE_CATEGORY
      value: "dev/application-name"
    - name: SUMO_SOURCE_NAME
      value: "application-name-dev"
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: config-volume
      mountPath: /fluentd/etc
  volumes:
  - name: varlog
    emptyDir: {}
  - name: config-volume
    configMap:
      name: fluentd-config