Last active
October 17, 2019 14:26
-
-
Save skriebel/8b80d51a75c99e64eb42 to your computer and use it in GitHub Desktop.
Logstash split field solution.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem: We have a log line that includes a perl class that we want to log the class | |
and method in their respected fields. An example class and method in perl: | |
Animal::Dog::bark | |
In this example, "bark" is the method. "Animal::Dog" is the class. | |
After some searching and hacking, I found a solution that works with Logstash 1.4.2 | |
Assume the input is "Animal::Dog::bark". | |
For completeness, I'm going to just add my entire configuration file I used for testing. | |
Comments are included and should explain what's going on. | |
input { | |
stdin { } | |
} | |
filter { | |
grok { | |
match => { "message" => "%{GREEDYDATA:api_class}" } | |
} | |
mutate { | |
# split the field on :: | |
split => ["api_class" , "::"] | |
# save the last element of the array as the api_method. | |
add_field => ["api_method", "%{[api_class][-1]}" ] | |
} | |
ruby { | |
# Go directly to the array and remove the last element. | |
code => "event['api_class'].pop()" | |
} | |
mutate { | |
# Join together whats left as the class name. | |
join => ["api_class", "::"] | |
} | |
} | |
output { | |
stdout { codec => rubydebug } | |
} | |
I tried to use mutate's remove_field to remove the last element of the array but it didn't work. | |
There are tickets created and possibly even a fix in the new version, however, this should continue | |
to work as long as the ruby filter is around. | |
The output: | |
{ | |
"message" => "Animal::Dog::bark", | |
"@version" => "1", | |
"@timestamp" => "2014-12-09T13:38:58.178Z", | |
"host" => "host.example.com", | |
"api_class" => "Animal::Dog", | |
"api_method" => "bark" | |
} | |
Hi. I came here from a stackoverflow answer: https://stackoverflow.com/a/46031524
I appreciate the Ruby code to understand some useful array operations :-)I think your use case could be solved with a simpler grok pattern though, the trick is to leverage the end anchor "$" in the regexp
grok { match => { "message" => "^%{GREEDYDATA:api_class}::%{WORD:api_method}$" } }
I totally forgot about this gist! I'm glad it's helping others and inspiring better solutions.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi. I came here from a stackoverflow answer: https://stackoverflow.com/a/46031524
I appreciate the Ruby code to understand some useful array operations :-)
I think your use case could be solved with a simpler grok pattern though, the trick is to leverage the end anchor "$" in the regexp
grok { match => { "message" => "^%{GREEDYDATA:api_class}::%{WORD:api_method}$" } }