Last active
July 7, 2023 00:25
-
-
Save bfritz/7831984 to your computer and use it in GitHub Desktop.
logstash base64 decode with ruby filter
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
$ cat base64_decode.conf | |
input { | |
stdin { } | |
} | |
filter { | |
grok { | |
match => ["message", "%{WORD:prefix} %{WORD:b64} %{WORD:suffix}"] | |
} | |
ruby { | |
init => "require 'base64'" | |
code => "event['b64_decoded'] = Base64.decode64(event['b64']) if event.include?('b64')" | |
} | |
} | |
output { | |
stdout { | |
codec => rubydebug | |
} | |
} | |
# messages: | |
# p bWlkZGxl s | |
# p middle s | |
# p s | |
$ echo "p bWlkZGxl s\np middle s\np s" | ./bin/logstash agent -f base64_decode.conf | |
[deprecated] I18n.enforce_available_locales will default to true in the future. If you really want to skip validation of your locale you can set I18n.enforce_available_locales = false to avoid this message. | |
Using milestone 1 filter plugin 'ruby'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. For more information on plugin milestones, see http://logstash.net/docs/1.2.3.dev/plugin-milestones {:level=>:warn} | |
{ | |
"message" => "p bWlkZGxl s", | |
"@timestamp" => "2013-12-07T04:09:16.662Z", | |
"@version" => "1", | |
"host" => "t61", | |
"prefix" => "p", | |
"b64" => "bWlkZGxl", | |
"suffix" => "s", | |
"b64_decoded" => "middle" | |
} | |
{ | |
"message" => "p middle s", | |
"@timestamp" => "2013-12-07T04:09:16.663Z", | |
"@version" => "1", | |
"host" => "t61", | |
"prefix" => "p", | |
"b64" => "middle", | |
"suffix" => "s", | |
"b64_decoded" => "\x9A']" | |
} | |
{ | |
"message" => "p s", | |
"@timestamp" => "2013-12-07T04:09:16.663Z", | |
"@version" => "1", | |
"host" => "t61", | |
"tags" => [ | |
[0] "_grokparsefailure" | |
] | |
} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
I am trying with below logstash conf file to decode base64 logs to send decoded data to elasticsearch but its throwing below error -
[2019-07-17T19:01:36,791][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `unpack1' for nil:NilClass
Logstash.conf file
input {
s3 {
access_key_id => "xxxxxxx"
bucket => "kinesis-datadog"
region => "us-east-1"
secret_access_key => "xxxxxxx8"
prefix => "2019/07/05"
type => "S3"
}
}
filter {
grok {
match => {"message" => "%{WORD:prefix} %{WORD:b64} %{WORD:suffix}"}
}
ruby {
init => "require 'base64'"
code => "event['b64_decoded'] = Base64.decode64(event['b64']) if event.include?('b64')"
#code => "event.set('[decodedBase64]', Base64.decode64(event.get('[base64_field]')))"
}
}
#output plugin, which has the ElasticSearch domain information
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => ".kibana-%{+YYYY.MM.dd}"
codec => rubydebug
#user => "elastic"
#password => "elastic"
}
}