Skip to content

Instantly share code, notes, and snippets.

@xemoe
Last active July 15, 2016 06:53
Show Gist options
  • Save xemoe/f344c1a8b0af2f8c8eb91cd98d6d4d77 to your computer and use it in GitHub Desktop.
Save xemoe/f344c1a8b0af2f8c8eb91cd98d6d4d77 to your computer and use it in GitHub Desktop.
SSH Files inputs
@xemoe
Copy link
Author

xemoe commented Jul 7, 2016

Add authorized_keys through ssh-copy-id

#!/bin/bash

[email protected]
IDENTITY_FILE=/opt/ssh_config/keys/mainbox/id_rsa
SSH_PASS=vagrant

sshpass -p${SSH_PASS} ssh-copy-id -o StrictHostKeyChecking=no -i ${IDENTITY_FILE} ${TARGET_HOST}

@xemoe
Copy link
Author

xemoe commented Jul 8, 2016

Logstash files forward configurations
---
logstash_configurations:
  - configurations_file: 10-files_forward.conf
    template_file: "opt/logstash/conf.d/10-files_forward.conf.j2"
    logstash_inputs:
      file: 
        from_file: /mnt/sshfs/mainbox/logs/syslog
        type: files_forward
        tags: ["mainbox_syslog"]
    logstash_outputs:
      file:
        to_file: syslog_172.100.0.200.log
    backup_directory: /backup/forwardlogs
    file_tag: "mainbox_syslog"
    active: true
Ref

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-file.html

TODOs
  • Use ansible role ansible-logstash_configurations
  • Create example logstash files forward configurations
  • Create 10-files_forward.conf.j2 template
  • Read file input from /mnt/sshfs/mainbox/logs/syslog
  • Add tags mainbox_syslog for file from /mnt/sshfs/mainbox/logs/syslog
  • Output file to /backup/forwardlogs/${DATE}/${FROMHOST}/${FILENAME}.${EXTENTIONS} for tag mainbox_syslog
  • Rotate logs in directory
  • Creat tasks
    • create logstash configurations file with ansible logstash
    • create supervisor_logstash supervisor configurations file
    • create log rotate configuration file
    • create log rotate schedule

@xemoe
Copy link
Author

xemoe commented Jul 13, 2016

10-files-forward.conf

input {
  file {
    path => "/mnt/sshfs/mainbox/logs/syslog"
    synced_path => "/opt/logstash/sincedb/mainbox.sincedb"
    start_position => "beginning"
    stat_interval => 1
    tags => [mainbox_syslog]
    type => "files_forward"
  }
}
output {
  if ([type] == "files_forward") {
    if "mainbox_syslog" in [tags] {
      file {
        path => "/backup/forwardlogs/%{+YYYY-MM-dd}/syslog_172.100.0.200.log"
        codec => { 
          line { format => "%{message}"} 
        }
      }
    }
  }
}

10-files-forward.conf.j2

{% if item.logstash_inputs.file is defined and item.logstash_inputs.file | length > 0 %}
{% if item.logstash_outputs.file is defined and item.logstash_outputs.file | length > 0 %}
input {
  file {
    path => "{{ item.logstash_inputs.file.from_file }}"
    synced_path => "{{ logstash_home }}/sincedb/{{ item.logstash_inputs.file.from_file | hash('sha1') }}.sincedb"
    start_position => "beginning"
    stat_interval => 1
    tags => {{ item.logstash_inputs.file.tags | to_yaml }}
    type => "{{ item.logstash_inputs.file.type | default("file_forward") }}"
  }
}
output {
    if ([type] == "{{ item.logstash_inputs.file.type }}") {
        if "{{ item.file_tag }}" in [tags] {
            file {
                path => "{{ item.backup_directory }}/%{+YYYY-MM-dd}/{{ item.logstash_outputs.file.to_file }}"
                codec => { line { format => "%{message}"} }
            }
        }
    }
}
{% endif %}
{% endif %}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment