Last active
July 15, 2016 06:53
-
-
Save xemoe/f344c1a8b0af2f8c8eb91cd98d6d4d77 to your computer and use it in GitHub Desktop.
SSH Files inputs
Logstash files forward configurations
---
logstash_configurations:
- configurations_file: 10-files_forward.conf
template_file: "opt/logstash/conf.d/10-files_forward.conf.j2"
logstash_inputs:
file:
from_file: /mnt/sshfs/mainbox/logs/syslog
type: files_forward
tags: ["mainbox_syslog"]
logstash_outputs:
file:
to_file: syslog_172.100.0.200.log
backup_directory: /backup/forwardlogs
file_tag: "mainbox_syslog"
active: true
Ref
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-file.html
TODOs
- Use ansible role ansible-logstash_configurations
- Create example logstash files forward configurations
- Create 10-files_forward.conf.j2 template
- Read file input from
/mnt/sshfs/mainbox/logs/syslog
- Add tags
mainbox_syslog
for file from/mnt/sshfs/mainbox/logs/syslog
- Output file to
/backup/forwardlogs/${DATE}/${FROMHOST}/${FILENAME}.${EXTENTIONS}
for tagmainbox_syslog
- Rotate logs in directory
- Creat tasks
- create logstash configurations file with ansible logstash
- create supervisor_logstash supervisor configurations file
- create log rotate configuration file
- create log rotate schedule
10-files-forward.conf
input {
file {
path => "/mnt/sshfs/mainbox/logs/syslog"
synced_path => "/opt/logstash/sincedb/mainbox.sincedb"
start_position => "beginning"
stat_interval => 1
tags => [mainbox_syslog]
type => "files_forward"
}
}
output {
if ([type] == "files_forward") {
if "mainbox_syslog" in [tags] {
file {
path => "/backup/forwardlogs/%{+YYYY-MM-dd}/syslog_172.100.0.200.log"
codec => {
line { format => "%{message}"}
}
}
}
}
}
10-files-forward.conf.j2
{% if item.logstash_inputs.file is defined and item.logstash_inputs.file | length > 0 %}
{% if item.logstash_outputs.file is defined and item.logstash_outputs.file | length > 0 %}
input {
file {
path => "{{ item.logstash_inputs.file.from_file }}"
synced_path => "{{ logstash_home }}/sincedb/{{ item.logstash_inputs.file.from_file | hash('sha1') }}.sincedb"
start_position => "beginning"
stat_interval => 1
tags => {{ item.logstash_inputs.file.tags | to_yaml }}
type => "{{ item.logstash_inputs.file.type | default("file_forward") }}"
}
}
output {
if ([type] == "{{ item.logstash_inputs.file.type }}") {
if "{{ item.file_tag }}" in [tags] {
file {
path => "{{ item.backup_directory }}/%{+YYYY-MM-dd}/{{ item.logstash_outputs.file.to_file }}"
codec => { line { format => "%{message}"} }
}
}
}
}
{% endif %}
{% endif %}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add authorized_keys through ssh-copy-id