Skip to content

Instantly share code, notes, and snippets.

@davidpfahler

davidpfahler/gist:81dfdd697c16549ac244

Created June 18, 2014 18:35
Show Gist options
  • Save davidpfahler/81dfdd697c16549ac244 to your computer and use it in GitHub Desktop.
Save davidpfahler/81dfdd697c16549ac244 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
Using milestone 1 input plugin 's3'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. For more information on plugin milestones, see http://logstash.net/docs/1.4.1/plugin-milestones {:level=>:warn}
Registering s3 input {:bucket=>"logs.appbusinesspodcast.com", :region_endpoint=>"us-east-1", :level=>:info}
Grok patterns path {:patterns_dir=>["/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/*"], :level=>:info}
Grok loading patterns from file {:path=>"/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/firewalls", :level=>:info}
Grok loading patterns from file {:path=>"/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/grok-patterns", :level=>:info}
Grok loading patterns from file {:path=>"/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/haproxy", :level=>:info}
Grok loading patterns from file {:path=>"/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/java", :level=>:info}
Grok loading patterns from file {:path=>"/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/junos", :level=>:info}
Grok loading patterns from file {:path=>"/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/linux-syslog", :level=>:info}
Grok loading patterns from file {:path=>"/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/mcollective", :level=>:info}
Grok loading patterns from file {:path=>"/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/mcollective-patterns", :level=>:info}
Grok loading patterns from file {:path=>"/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/mongodb", :level=>:info}
Grok loading patterns from file {:path=>"/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/nagios", :level=>:info}
Grok loading patterns from file {:path=>"/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/postgresql", :level=>:info}
Grok loading patterns from file {:path=>"/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/redis", :level=>:info}
Grok loading patterns from file {:path=>"/Users/djpfahler/localhost/logstash_stats/logstash-1.4.1/lib/logstash/filters/../../../patterns/ruby", :level=>:info}
Match data {:match=>{"message"=>"%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:loadbalancer} %{IP:client_ip}:%{NUMBER:client_port:int} %{IP:backend_ip}:%{NUMBER:backend_port:int} %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{NUMBER:elb_status_code:int} %{NUMBER:backend_status_code:int} %{NUMBER:received_bytes:int} %{NUMBER:sent_bytes:int} %{QS:request}"}, :level=>:info}
Grok compile {:field=>"message", :patterns=>["%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:loadbalancer} %{IP:client_ip}:%{NUMBER:client_port:int} %{IP:backend_ip}:%{NUMBER:backend_port:int} %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{NUMBER:elb_status_code:int} %{NUMBER:backend_status_code:int} %{NUMBER:received_bytes:int} %{NUMBER:sent_bytes:int} %{QS:request}"], :level=>:info}
Pipeline started {:level=>:info}
log4j, [2014-06-18T20:34:45.476] INFO: org.elasticsearch.node: [logstash-authority.local-48964-2192] version[1.1.1], pid[48964], build[f1585f0/2014-04-16T14:27:12Z]
log4j, [2014-06-18T20:34:45.476] INFO: org.elasticsearch.node: [logstash-authority.local-48964-2192] initializing ...
log4j, [2014-06-18T20:34:45.487] INFO: org.elasticsearch.plugins: [logstash-authority.local-48964-2192] loaded [], sites []
log4j, [2014-06-18T20:34:47.282] INFO: org.elasticsearch.node: [logstash-authority.local-48964-2192] initialized
log4j, [2014-06-18T20:34:47.282] INFO: org.elasticsearch.node: [logstash-authority.local-48964-2192] starting ...
log4j, [2014-06-18T20:34:47.381] INFO: org.elasticsearch.transport: [logstash-authority.local-48964-2192] bound_address {inet[/0:0:0:0:0:0:0:0%0:9300]}, publish_address {inet[/192.168.2.109:9300]}
log4j, [2014-06-18T20:34:51.158] WARN: org.elasticsearch.discovery.zen.ping.unicast: [logstash-authority.local-48964-2192] failed to send ping to [[#zen_unicast_1#][authority.local][inet[/192.168.59.103:49153]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/192.168.59.103:49153]][discovery/zen/unicast] request_id [0] timed out after [3751ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
log4j, [2014-06-18T20:34:55.649] WARN: org.elasticsearch.discovery.zen.ping.unicast: [logstash-authority.local-48964-2192] failed to send ping to [[#zen_unicast_1#][authority.local][inet[/192.168.59.103:49153]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/192.168.59.103:49153]][discovery/zen/unicast] request_id [3] timed out after [3751ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
log4j, [2014-06-18T20:35:00.151] WARN: org.elasticsearch.discovery.zen.ping.unicast: [logstash-authority.local-48964-2192] failed to send ping to [[#zen_unicast_1#][authority.local][inet[/192.168.59.103:49153]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/192.168.59.103:49153]][discovery/zen/unicast] request_id [6] timed out after [3750ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
log4j, [2014-06-18T20:35:09.161] WARN: org.elasticsearch.discovery.zen.ping.unicast: [logstash-authority.local-48964-2192] failed to send ping to [[#zen_unicast_1#][authority.local][inet[/192.168.59.103:49153]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/192.168.59.103:49153]][discovery/zen/unicast] request_id [12] timed out after [3752ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
log4j, [2014-06-18T20:35:13.662] WARN: org.elasticsearch.discovery.zen.ping.unicast: [logstash-authority.local-48964-2192] failed to send ping to [[#zen_unicast_1#][authority.local][inet[/192.168.59.103:49153]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/192.168.59.103:49153]][discovery/zen/unicast] request_id [15] timed out after [3751ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
log4j, [2014-06-18T20:35:17.386] WARN: org.elasticsearch.discovery: [logstash-authority.local-48964-2192] waited for 30s and no initial state was set by the discovery
log4j, [2014-06-18T20:35:17.386] INFO: org.elasticsearch.discovery: [logstash-authority.local-48964-2192] elasticsearch/tPUar-Q4Q3628REmbe81_w
log4j, [2014-06-18T20:35:17.393] INFO: org.elasticsearch.node: [logstash-authority.local-48964-2192] started
New Elasticsearch output {:cluster=>nil, :host=>"192.168.59.103", :port=>49153, :embedded=>false, :protocol=>"node", :level=>:info}
Automatic template management enabled {:manage_template=>"true", :level=>:info}
Using mapping template {:template=>"{ \"template\" : \"logstash-*\", \"settings\" : { \"index.refresh_interval\" : \"5s\" }, \"mappings\" : { \"_default_\" : { \"_all\" : {\"enabled\" : true}, \"dynamic_templates\" : [ { \"string_fields\" : { \"match\" : \"*\", \"match_mapping_type\" : \"string\", \"mapping\" : { \"type\" : \"string\", \"index\" : \"analyzed\", \"omit_norms\" : true, \"fields\" : { \"raw\" : {\"type\": \"string\", \"index\" : \"not_analyzed\", \"ignore_above\" : 256} } } } } ], \"properties\" : { \"@version\": { \"type\": \"string\", \"index\": \"not_analyzed\" }, \"geoip\" : { \"type\" : \"object\", \"dynamic\": true, \"path\": \"full\", \"properties\" : { \"location\" : { \"type\" : \"geo_point\" } } } } } }}", :level=>:info}
log4j, [2014-06-18T20:35:18.164] WARN: org.elasticsearch.discovery.zen.ping.unicast: [logstash-authority.local-48964-2192] failed to send ping to [[#zen_unicast_1#][authority.local][inet[/192.168.59.103:49153]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/192.168.59.103:49153]][discovery/zen/unicast] request_id [18] timed out after [3751ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
log4j, [2014-06-18T20:35:22.669] WARN: org.elasticsearch.discovery.zen.ping.unicast: [logstash-authority.local-48964-2192] failed to send ping to [[#zen_unicast_1#][authority.local][inet[/192.168.59.103:49153]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/192.168.59.103:49153]][discovery/zen/unicast] request_id [21] timed out after [3752ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
log4j, [2014-06-18T20:35:27.174] WARN: org.elasticsearch.discovery.zen.ping.unicast: [logstash-authority.local-48964-2192] failed to send ping to [[#zen_unicast_1#][authority.local][inet[/192.168.59.103:49153]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/192.168.59.103:49153]][discovery/zen/unicast] request_id [24] timed out after [3751ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment