fluentd-install.yaml

fluentd-install.yaml

---
- name: playbook to install fluentd agent
  hosts: all
  become: true
  vars:
    log_server: LOG_SERVER.EXAMPLE.COM
    log_port: 10405  # Defaults to 514 if not set

  tasks:

  - name: Get latest package installer
    get_url:
      url: https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh
      dest: /usr/local/share/install-redhat-td-agent3.sh
      mode: 0755

  - name: Run installer
    command: "/bin/bash /usr/local/share/install-redhat-td-agent3.sh"

  - name: Install fluent-plugin-kubernetes
    command: /opt/td-agent/embedded/bin/gem install fluent-plugin-kubernetes

  - name: Create config directory
    file:
      path: /etc/td-agent/config.d
      recurse: yes
      state: directory
      owner: td-agent
      group: td-agent

  - name: Create log directory
    file:
      path: /var/log/td-agent/tmp
      recurse: yes
      state: directory
      owner: td-agent
      group: td-agent
    
  - name: Create config file
    copy:
      dest: /etc/sysconfig/td-agent
      content: 'DAEMON_ARGS=\nTD_AGENT_ARGS="/usr/sbin/td-agent --log /var/log/td-agent/td-agent.log --use-v1-config\n'

  - name: Modify td-agent.conf
    lineinfile:
      path: /etc/td-agent/td-agent.conf
      state: present
      insertafter: EOF
      line: "@include config.d/*.conf"

  - name: Create kubernetes.conf file
    template:
      dest: /etc/td-agent/config.d/kubernetes.conf
      src: 'kubernetes.conf.j2'

  - name: Enable and Start td-agent
    service:
      name: td-agent
      state: started
      enabled: yes

kubernetes.conf.j2

<source>
  type tail
  path /var/lib/docker/containers/*/*-json.log
  pos_file /var/log/td-agent/tmp/fluentd-docker.pos
  time_format %Y-%m-%dT%H:%M:%S
  tag docker.*
  format json
  read_from_head true
</source>

<match docker.var.lib.docker.containers.*.*.log>
  type kubernetes
  container_id ${tag_parts[5]}
  tag docker.${name}
</match>

<match kubernetes>
  type copy
  <store>
    type forward
    send_timeout 60s
    recover_wait 10s
    heartbeat_interval 1s
    phi_threshold 16
    hard_timeout 60s
    #log_level trace
    log_level warn
    require_ack_response true
    heartbeat_type tcp
    <server>
      name OCP_logs
      host {{ log_server }}
      port {{ log_port | default(514) }}
      weight 60
    </server>

    <secondary>
      type file
      path /var/log/td-agent/forward-failed
    </secondary>
  </store>

  <store>
    type file
    path /var/log/td-agent/containers.log
    time_slice_format %Y%m%d
    time_slice_wait 10m
    time_format %Y%m%dT%H%M%S%z
    utc
  </store>
</match>

This gist is an ansible playbook to install the latest version of td-agent (Fluentd forwarder) It creates a kubernetes.conf file so that this can be used to track OCP docker pods. When run, it requires a log_server and log_port to be set. The playbook will need to be modified to accommodate that.

Removed the creates line as it doesn't seem to do what I wanted.