Skip to content

Instantly share code, notes, and snippets.

@CrackerJackMack
Created January 23, 2015 02:52
Show Gist options
  • Save CrackerJackMack/e89dc7459669e6264441 to your computer and use it in GitHub Desktop.
Save CrackerJackMack/e89dc7459669e6264441 to your computer and use it in GitHub Desktop.
split json top level arrays into separate events
--- logstash-1.4.2/lib/logstash/codecs/json.rb 2014-06-24 08:08:27.000000000 -0500
+++ logstash-1.4.2/lib/logstash/codecs/json_split.rb 2015-01-17 23:17:59.073263327 -0600
@@ -7,8 +7,8 @@
# full JSON messages. If you are streaming JSON messages delimited
# by '\n' then see the `json_lines` codec.
# Encoding will result in a single JSON string.
-class LogStash::Codecs::JSON < LogStash::Codecs::Base
- config_name "json"
+class LogStash::Codecs::JSONSplit < LogStash::Codecs::Base
+ config_name "json_split"
milestone 3
@@ -33,7 +33,18 @@
def decode(data)
data = @converter.convert(data)
begin
- yield LogStash::Event.new(JSON.parse(data))
+ @logger.debug("json decoding")
+ parsed_data = JSON.parse(data)
+ @logger.debug("json decoded", :data => j)
+ if parsed_data.kind_of?(Array)
+ @logger.info("JSON array found, processing multiple events")
+ parsed_data.each do |event|
+ yield LogStash::Event.new(event)
+ end
+ else
+ @logger.info("JSON object found, single event")
+ yield LogStash::Event.new(parsed_data)
+ end
rescue JSON::ParserError => e
@logger.info("JSON parse failure. Falling back to plain-text", :error => e, :data => data)
yield LogStash::Event.new("message" => data)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment