Skip to content

Instantly share code, notes, and snippets.

@FilBot3

FilBot3/logstash_5.6.3_CSV_parse_failure.sh

Created October 11, 2017 19:00
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save FilBot3/28494bea8499f347bb03fa9078f38d72 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save FilBot3/28494bea8499f347bb03fa9078f38d72 to your computer and use it in GitHub Desktop.
Download ZIP
Failure of LogStash 5.6.3 starting and parsing CSV Files.
Raw
logstash_5.6.3_CSV_parse_failure.sh
predatorian3@device01 [13:47:08] ~/Downloads/logstash-5.6.3
$ bin/logstash -f logstash.conf
Sending Logstash's logs to /Users/predatorian3/Downloads/logstash-5.6.3/logs which is now configured via log4j2.properties
[2017-10-11T13:47:57,985][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/Users/predatorian3/Downloads/logstash-5.6.3/modules/fb_apache/configuration"}
[2017-10-11T13:47:57,990][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/Users/predatorian3/Downloads/logstash-5.6.3/modules/netflow/configuration"}
[2017-10-11T13:47:58,312][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-10-11T13:47:58,525][INFO ][logstash.pipeline ] Pipeline main started
[2017-10-11T13:47:58,600][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
^C[2017-10-11T13:49:24,412][WARN ][logstash.runner ] SIGINT received. Shutting down the agent.
[2017-10-11T13:49:24,422][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
predatorian3@device01 [13:49:25] ~/Downloads/logstash-5.6.3
$ bin/logstash -f logstash.conf --log.level debug
Sending Logstash's logs to /Users/predatorian3/Downloads/logstash-5.6.3/logs which is now configured via log4j2.properties
[2017-10-11T13:50:00,770][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/Users/predatorian3/Downloads/logstash-5.6.3/modules/fb_apache/configuration"}
[2017-10-11T13:50:00,775][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x5464eb28 @kibana_version_parts=["5", "6", "0"], @module_name="fb_apache", @directory="/Users/predatorian3/Downloads/logstash-5.6.3/modules/fb_apache/configuration">}
[2017-10-11T13:50:00,777][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/Users/predatorian3/Downloads/logstash-5.6.3/modules/netflow/configuration"}
[2017-10-11T13:50:00,778][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x3998a6ce @kibana_version_parts=["5", "6", "0"], @module_name="netflow", @directory="/Users/predatorian3/Downloads/logstash-5.6.3/modules/netflow/configuration">}
[2017-10-11T13:50:00,790][DEBUG][logstash.runner ] -------- Logstash Settings (* means modified) ---------
[2017-10-11T13:50:00,790][DEBUG][logstash.runner ] node.name: "device01"
[2017-10-11T13:50:00,790][DEBUG][logstash.runner ] *path.config: "logstash.conf"
[2017-10-11T13:50:00,790][DEBUG][logstash.runner ] path.data: "/Users/predatorian3/Downloads/logstash-5.6.3/data"
[2017-10-11T13:50:00,791][DEBUG][logstash.runner ] modules.cli: []
[2017-10-11T13:50:00,791][DEBUG][logstash.runner ] modules: []
[2017-10-11T13:50:00,791][DEBUG][logstash.runner ] modules_setup: false
[2017-10-11T13:50:00,791][DEBUG][logstash.runner ] config.test_and_exit: false
[2017-10-11T13:50:00,791][DEBUG][logstash.runner ] config.reload.automatic: false
[2017-10-11T13:50:00,791][DEBUG][logstash.runner ] config.support_escapes: false
[2017-10-11T13:50:00,791][DEBUG][logstash.runner ] config.reload.interval: 3
[2017-10-11T13:50:00,792][DEBUG][logstash.runner ] metric.collect: true
[2017-10-11T13:50:00,792][DEBUG][logstash.runner ] pipeline.id: "main"
[2017-10-11T13:50:00,792][DEBUG][logstash.runner ] pipeline.system: false
[2017-10-11T13:50:00,793][DEBUG][logstash.runner ] pipeline.workers: 4
[2017-10-11T13:50:00,793][DEBUG][logstash.runner ] pipeline.output.workers: 1
[2017-10-11T13:50:00,793][DEBUG][logstash.runner ] pipeline.batch.size: 125
[2017-10-11T13:50:00,793][DEBUG][logstash.runner ] pipeline.batch.delay: 5
[2017-10-11T13:50:00,794][DEBUG][logstash.runner ] pipeline.unsafe_shutdown: false
[2017-10-11T13:50:00,794][DEBUG][logstash.runner ] path.plugins: []
[2017-10-11T13:50:00,794][DEBUG][logstash.runner ] config.debug: false
[2017-10-11T13:50:00,794][DEBUG][logstash.runner ] *log.level: "debug" (default: "info")
[2017-10-11T13:50:00,794][DEBUG][logstash.runner ] version: false
[2017-10-11T13:50:00,795][DEBUG][logstash.runner ] help: false
[2017-10-11T13:50:00,795][DEBUG][logstash.runner ] log.format: "plain"
[2017-10-11T13:50:00,795][DEBUG][logstash.runner ] http.host: "127.0.0.1"
[2017-10-11T13:50:00,795][DEBUG][logstash.runner ] http.port: 9600..9700
[2017-10-11T13:50:00,795][DEBUG][logstash.runner ] http.environment: "production"
[2017-10-11T13:50:00,795][DEBUG][logstash.runner ] queue.type: "memory"
[2017-10-11T13:50:00,795][DEBUG][logstash.runner ] queue.drain: false
[2017-10-11T13:50:00,795][DEBUG][logstash.runner ] queue.page_capacity: 262144000
[2017-10-11T13:50:00,795][DEBUG][logstash.runner ] queue.max_bytes: 1073741824
[2017-10-11T13:50:00,796][DEBUG][logstash.runner ] queue.max_events: 0
[2017-10-11T13:50:00,796][DEBUG][logstash.runner ] queue.checkpoint.acks: 1024
[2017-10-11T13:50:00,796][DEBUG][logstash.runner ] queue.checkpoint.writes: 1024
[2017-10-11T13:50:00,857][DEBUG][logstash.runner ] queue.checkpoint.interval: 1000
[2017-10-11T13:50:00,857][DEBUG][logstash.runner ] dead_letter_queue.enable: false
[2017-10-11T13:50:00,857][DEBUG][logstash.runner ] dead_letter_queue.max_bytes: 1073741824
[2017-10-11T13:50:00,857][DEBUG][logstash.runner ] slowlog.threshold.warn: -1
[2017-10-11T13:50:00,858][DEBUG][logstash.runner ] slowlog.threshold.info: -1
[2017-10-11T13:50:00,858][DEBUG][logstash.runner ] slowlog.threshold.debug: -1
[2017-10-11T13:50:00,858][DEBUG][logstash.runner ] slowlog.threshold.trace: -1
[2017-10-11T13:50:00,858][DEBUG][logstash.runner ] path.queue: "/Users/predatorian3/Downloads/logstash-5.6.3/data/queue"
[2017-10-11T13:50:00,858][DEBUG][logstash.runner ] path.dead_letter_queue: "/Users/predatorian3/Downloads/logstash-5.6.3/data/dead_letter_queue"
[2017-10-11T13:50:00,858][DEBUG][logstash.runner ] path.settings: "/Users/predatorian3/Downloads/logstash-5.6.3/config"
[2017-10-11T13:50:00,859][DEBUG][logstash.runner ] path.logs: "/Users/predatorian3/Downloads/logstash-5.6.3/logs"
[2017-10-11T13:50:00,859][DEBUG][logstash.runner ] --------------- Logstash Settings -------------------
[2017-10-11T13:50:00,899][DEBUG][logstash.agent ] Agent: Configuring metric collection
[2017-10-11T13:50:00,902][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
[2017-10-11T13:50:00,943][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
[2017-10-11T13:50:00,998][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
[2017-10-11T13:50:01,011][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
[2017-10-11T13:50:01,024][DEBUG][logstash.agent ] Reading config file {:config_file=>"/Users/predatorian3/Downloads/logstash-5.6.3/logstash.conf"}
[2017-10-11T13:50:01,146][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"file", :type=>"input", :class=>LogStash::Inputs::File}
[2017-10-11T13:50:01,162][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"plain", :type=>"codec", :class=>LogStash::Codecs::Plain}
[2017-10-11T13:50:01,166][DEBUG][logstash.codecs.plain ] config LogStash::Codecs::Plain/@id = "plain_1c3d8276-8623-4b8b-924e-79d46de2ddc8"
[2017-10-11T13:50:01,166][DEBUG][logstash.codecs.plain ] config LogStash::Codecs::Plain/@enable_metric = true
[2017-10-11T13:50:01,166][DEBUG][logstash.codecs.plain ] config LogStash::Codecs::Plain/@charset = "UTF-8"
[2017-10-11T13:50:01,168][DEBUG][logstash.inputs.file ] config LogStash::Inputs::File/@type = "csv"
[2017-10-11T13:50:01,168][DEBUG][logstash.inputs.file ] config LogStash::Inputs::File/@path = ["/Users/predatorian3/Downloads/sec_vulns/sec_vulns.csv"]
[2017-10-11T13:50:01,168][DEBUG][logstash.inputs.file ] config LogStash::Inputs::File/@start_position = "beginning"
[2017-10-11T13:50:01,169][DEBUG][logstash.inputs.file ] config LogStash::Inputs::File/@tags = ["security", "stig"]
[2017-10-11T13:50:01,169][DEBUG][logstash.inputs.file ] config LogStash::Inputs::File/@id = "b4dd61f25bc420e12f9c29192bd7d963c625cb6b-1"
[2017-10-11T13:50:01,170][DEBUG][logstash.inputs.file ] config LogStash::Inputs::File/@enable_metric = true
[2017-10-11T13:50:01,170][DEBUG][logstash.inputs.file ] config LogStash::Inputs::File/@codec = <LogStash::Codecs::Plain id=>"plain_1c3d8276-8623-4b8b-924e-79d46de2ddc8", enable_metric=>true, charset=>"UTF-8">
[2017-10-11T13:50:01,170][DEBUG][logstash.inputs.file ] config LogStash::Inputs::File/@add_field = {}
[2017-10-11T13:50:01,170][DEBUG][logstash.inputs.file ] config LogStash::Inputs::File/@stat_interval = 1
[2017-10-11T13:50:01,204][DEBUG][logstash.inputs.file ] config LogStash::Inputs::File/@discover_interval = 15
[2017-10-11T13:50:01,205][DEBUG][logstash.inputs.file ] config LogStash::Inputs::File/@sincedb_write_interval = 15
[2017-10-11T13:50:01,205][DEBUG][logstash.inputs.file ] config LogStash::Inputs::File/@delimiter = "\n"
[2017-10-11T13:50:01,205][DEBUG][logstash.inputs.file ] config LogStash::Inputs::File/@close_older = 3600
[2017-10-11T13:50:01,255][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"csv", :type=>"filter", :class=>LogStash::Filters::CSV}
[2017-10-11T13:50:01,264][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@autodetect_column_names = true
[2017-10-11T13:50:01,264][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@id = "b4dd61f25bc420e12f9c29192bd7d963c625cb6b-2"
[2017-10-11T13:50:01,265][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@enable_metric = true
[2017-10-11T13:50:01,265][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@add_tag = []
[2017-10-11T13:50:01,265][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@remove_tag = []
[2017-10-11T13:50:01,265][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@add_field = {}
[2017-10-11T13:50:01,265][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@remove_field = []
[2017-10-11T13:50:01,266][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@periodic_flush = false
[2017-10-11T13:50:01,266][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@source = "message"
[2017-10-11T13:50:01,266][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@columns = []
[2017-10-11T13:50:01,266][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@separator = ","
[2017-10-11T13:50:01,266][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@quote_char = "\""
[2017-10-11T13:50:01,266][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@autogenerate_column_names = true
[2017-10-11T13:50:01,267][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@skip_empty_columns = false
[2017-10-11T13:50:01,267][DEBUG][logstash.filters.csv ] config LogStash::Filters::CSV/@convert = {}
[2017-10-11T13:50:01,276][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"file", :type=>"output", :class=>LogStash::Outputs::File}
[2017-10-11T13:50:01,303][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"json_lines", :type=>"codec", :class=>LogStash::Codecs::JSONLines}
[2017-10-11T13:50:01,306][DEBUG][logstash.codecs.jsonlines] config LogStash::Codecs::JSONLines/@id = "json_lines_04af64a2-bca0-4f9a-a5ed-02734bad21a4"
[2017-10-11T13:50:01,307][DEBUG][logstash.codecs.jsonlines] config LogStash::Codecs::JSONLines/@enable_metric = true
[2017-10-11T13:50:01,307][DEBUG][logstash.codecs.jsonlines] config LogStash::Codecs::JSONLines/@charset = "UTF-8"
[2017-10-11T13:50:01,307][DEBUG][logstash.codecs.jsonlines] config LogStash::Codecs::JSONLines/@delimiter = "\n"
[2017-10-11T13:50:01,308][DEBUG][logstash.outputs.file ] config LogStash::Outputs::File/@path = "/Users/predatorian3/Downloads/sec_vulns/output.txt"
[2017-10-11T13:50:01,309][DEBUG][logstash.outputs.file ] config LogStash::Outputs::File/@id = "b4dd61f25bc420e12f9c29192bd7d963c625cb6b-3"
[2017-10-11T13:50:01,309][DEBUG][logstash.outputs.file ] config LogStash::Outputs::File/@enable_metric = true
[2017-10-11T13:50:01,309][DEBUG][logstash.outputs.file ] config LogStash::Outputs::File/@codec = <LogStash::Codecs::JSONLines id=>"json_lines_04af64a2-bca0-4f9a-a5ed-02734bad21a4", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">
[2017-10-11T13:50:01,309][DEBUG][logstash.outputs.file ] config LogStash::Outputs::File/@workers = 1
[2017-10-11T13:50:01,309][DEBUG][logstash.outputs.file ] config LogStash::Outputs::File/@flush_interval = 2
[2017-10-11T13:50:01,310][DEBUG][logstash.outputs.file ] config LogStash::Outputs::File/@gzip = false
[2017-10-11T13:50:01,335][DEBUG][logstash.outputs.file ] config LogStash::Outputs::File/@filename_failure = "_filepath_failures"
[2017-10-11T13:50:01,335][DEBUG][logstash.outputs.file ] config LogStash::Outputs::File/@create_if_deleted = true
[2017-10-11T13:50:01,335][DEBUG][logstash.outputs.file ] config LogStash::Outputs::File/@dir_mode = -1
[2017-10-11T13:50:01,336][DEBUG][logstash.outputs.file ] config LogStash::Outputs::File/@file_mode = -1
[2017-10-11T13:50:01,353][DEBUG][logstash.agent ] starting agent
[2017-10-11T13:50:01,386][DEBUG][logstash.agent ] starting pipeline {:id=>"main"}
[2017-10-11T13:50:01,392][DEBUG][logstash.filters.csv ] CSV parsing options {:col_sep=>",", :quote_char=>"\""}
[2017-10-11T13:50:01,395][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-10-11T13:50:01,409][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"undefined method `to_hash' for []:Array", "backtrace"=>["(eval):41:in `filter_func'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:398:in `filter_batch'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:379:in `worker_loop'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}
[2017-10-11T13:50:01,421][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"undefined method `to_hash' for []:Array", "backtrace"=>["(eval):41:in `filter_func'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:398:in `filter_batch'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:379:in `worker_loop'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}
[2017-10-11T13:50:01,427][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"undefined method `to_hash' for []:Array", "backtrace"=>["(eval):41:in `filter_func'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:398:in `filter_batch'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:379:in `worker_loop'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}
[2017-10-11T13:50:01,427][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"undefined method `to_hash' for []:Array", "backtrace"=>["(eval):41:in `filter_func'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:398:in `filter_batch'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:379:in `worker_loop'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}
[2017-10-11T13:50:01,590][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method `to_hash' for []:Array>, :backtrace=>["(eval):41:in `filter_func'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:398:in `filter_batch'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:379:in `worker_loop'", "/Users/predatorian3/Downloads/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}
predatorian3@device01 [13:50:01] ~/Downloads/logstash-5.6.3
$ cat logstash.conf
# Input sources for the LogStash Agent/Server
input {
file {
type => "csv"
path => [
"/Users/predatorian3/Downloads/sec_vulns/sec_vulns.csv"
]
start_position => "beginning"
tags => ["security", "stig"]
}
}
filter {
if [type] =~ "csv" {
csv {
autodetect_column_names => true
}
}
}
# Where is the data going? Defined here.
output {
#stdout { codec => rubydebug }
#elasticsearch {
# hosts => [ "localhost:9200" ]
#}
file {
path => "/Users/predatorian3/Downloads/sec_vulns/output.txt"
}
}
predatorian3@device01 [13:50:36] ~/Downloads/logstash-5.6.3
$ java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment