Skip to content

Instantly share code, notes, and snippets.

@skriebel
Last active October 17, 2019 14:26
Show Gist options
  • Save skriebel/8b80d51a75c99e64eb42 to your computer and use it in GitHub Desktop.
Save skriebel/8b80d51a75c99e64eb42 to your computer and use it in GitHub Desktop.
Logstash split field solution.
Problem: We have a log line that includes a perl class that we want to log the class
and method in their respected fields. An example class and method in perl:
Animal::Dog::bark
In this example, "bark" is the method. "Animal::Dog" is the class.
After some searching and hacking, I found a solution that works with Logstash 1.4.2
Assume the input is "Animal::Dog::bark".
For completeness, I'm going to just add my entire configuration file I used for testing.
Comments are included and should explain what's going on.
input {
stdin { }
}
filter {
grok {
match => { "message" => "%{GREEDYDATA:api_class}" }
}
mutate {
# split the field on ::
split => ["api_class" , "::"]
# save the last element of the array as the api_method.
add_field => ["api_method", "%{[api_class][-1]}" ]
}
ruby {
# Go directly to the array and remove the last element.
code => "event['api_class'].pop()"
}
mutate {
# Join together whats left as the class name.
join => ["api_class", "::"]
}
}
output {
stdout { codec => rubydebug }
}
I tried to use mutate's remove_field to remove the last element of the array but it didn't work.
There are tickets created and possibly even a fix in the new version, however, this should continue
to work as long as the ruby filter is around.
The output:
{
"message" => "Animal::Dog::bark",
"@version" => "1",
"@timestamp" => "2014-12-09T13:38:58.178Z",
"host" => "host.example.com",
"api_class" => "Animal::Dog",
"api_method" => "bark"
}
@skriebel
Copy link
Author

Hi. I came here from a stackoverflow answer: https://stackoverflow.com/a/46031524
I appreciate the Ruby code to understand some useful array operations :-)

I think your use case could be solved with a simpler grok pattern though, the trick is to leverage the end anchor "$" in the regexp

grok { match => { "message" => "^%{GREEDYDATA:api_class}::%{WORD:api_method}$" } }

I totally forgot about this gist! I'm glad it's helping others and inspiring better solutions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment