SSH Files inputs

ssh_files_inputs.md

SSH Files inputs

Depend

• ansible-ssh_config
• ansible-mount_sshfs
• ansible-logstash_configurations

Logstash files forward configurations

---
logstash_configurations:
  - configurations_file: 10-files_forward.conf
    template_file: "opt/logstash/conf.d/10-files_forward.conf.j2"
    logstash_inputs:
      file: 
        from_file: /mnt/sshfs/mainbox/logs/syslog
        type: files_forward
        tags: ["mainbox_syslog"]
    logstash_outputs:
      file:
        to_file: syslog_172.100.0.200.log
    backup_directory: /backup/forwardlogs
    file_tag: "mainbox_syslog"
    active: true

Ref

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-file.html

TODOs

• Use ansible role ansible-logstash_configurations
• Create example logstash files forward configurations
• Create 10-files_forward.conf.j2 template
• Read file input from /mnt/sshfs/mainbox/logs/syslog
• Add tags mainbox_syslog for file from /mnt/sshfs/mainbox/logs/syslog
• Output file to /backup/forwardlogs/${DATE}/${FROMHOST}/${FILENAME}.${EXTENTIONS} for tag mainbox_syslog
• Rotate logs in directory
• Creat tasks
  • create logstash configurations file with ansible logstash
  • create supervisor_logstash supervisor configurations file
  • create log rotate configuration file
  • create log rotate schedule

10-files-forward.conf

input {
  file {
    path => "/mnt/sshfs/mainbox/logs/syslog"
    synced_path => "/opt/logstash/sincedb/mainbox.sincedb"
    start_position => "beginning"
    stat_interval => 1
    tags => [mainbox_syslog]
    type => "files_forward"
  }
}
output {
  if ([type] == "files_forward") {
    if "mainbox_syslog" in [tags] {
      file {
        path => "/backup/forwardlogs/%{+YYYY-MM-dd}/syslog_172.100.0.200.log"
        codec => { 
          line { format => "%{message}"} 
        }
      }
    }
  }
}

---

10-files-forward.conf.j2

{% if item.logstash_inputs.file is defined and item.logstash_inputs.file | length > 0 %}
{% if item.logstash_outputs.file is defined and item.logstash_outputs.file | length > 0 %}
input {
  file {
    path => "{{ item.logstash_inputs.file.from_file }}"
    synced_path => "{{ logstash_home }}/sincedb/{{ item.logstash_inputs.file.from_file | hash('sha1') }}.sincedb"
    start_position => "beginning"
    stat_interval => 1
    tags => {{ item.logstash_inputs.file.tags | to_yaml }}
    type => "{{ item.logstash_inputs.file.type | default("file_forward") }}"
  }
}
output {
    if ([type] == "{{ item.logstash_inputs.file.type }}") {
        if "{{ item.file_tag }}" in [tags] {
            file {
                path => "{{ item.backup_directory }}/%{+YYYY-MM-dd}/{{ item.logstash_outputs.file.to_file }}"
                codec => { line { format => "%{message}"} }
            }
        }
    }
}
{% endif %}
{% endif %}