Skip to content

Instantly share code, notes, and snippets.

@cwarden

cwarden/README.md

Created January 31, 2012 21:53
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save cwarden/1713225 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save cwarden/1713225 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
README.md

Starting logstash like this:

java -jar /tmp/logstash-1.1.0-monolithic.jar agent -f mylogstash.conf -- web --backend elasticsearch:///?local 2>&1 | tee logstash.out

$ curl http://localhost:9200/
{
  "ok" : true,
  "name" : "Man-Elephant",
  "version" : {
    "number" : "0.18.7",
    "snapshot_build" : false
  },
  "tagline" : "You Know, for Search",
  "cover" : "DON'T PANIC",
  "quote" : {
    "book" : "The Restaurant at the End of the Universe",
    "chapter" : "Chapter 17",
    "text1" : "I am the main Dish of the Day. May I interest you in parts of my body?"
  }
}
Raw
logstash.out
I, [2012-01-31T13:54:21.615000 #25218] INFO -- : Using beta plugin 'file'. For more information about plugin statuses, see http://logstash.net/docs/1.1.0/plugin-status {"timestamp":"2012-01-31T13:54:21.595000 -0800","message":"Using beta plugin 'file'. For more information about plugin statuses, see http://logstash.net/docs/1.1.0/plugin-status ","level":"info"}
I, [2012-01-31T13:54:21.642000 #25218] INFO -- : Using beta plugin 'file'. For more information about plugin statuses, see http://logstash.net/docs/1.1.0/plugin-status {"timestamp":"2012-01-31T13:54:21.641000 -0800","message":"Using beta plugin 'file'. For more information about plugin statuses, see http://logstash.net/docs/1.1.0/plugin-status ","level":"info"}
I, [2012-01-31T13:54:21.656000 #25218] INFO -- : Using beta plugin 'file'. For more information about plugin statuses, see http://logstash.net/docs/1.1.0/plugin-status {"timestamp":"2012-01-31T13:54:21.655000 -0800","message":"Using beta plugin 'file'. For more information about plugin statuses, see http://logstash.net/docs/1.1.0/plugin-status ","level":"info"}
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-1.3.4/lib/rack/backports/uri/common_192.rb:53 warning: already initialized constant WFKV_
W, [2012-01-31T13:54:22.180000 #25218] WARN -- : failed to open /var/log/php5-fpm.log: Permission denied - /var/log/php5-fpm.log {"timestamp":"2012-01-31T13:54:22.175000 -0800","message":"failed to open /var/log/php5-fpm.log: Permission denied - /var/log/php5-fpm.log","level":"warn"}
log4j, [2012-01-31T13:54:56.210] WARN: org.elasticsearch.discovery: [Vincente] waited for 30s and no initial state was set by the discovery
Mizuno 0.5.0 (Jetty 8.0.y.z-SNAPSHOT) listening on 0.0.0.0:9292
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /search HTTP/1.1" 200 2348 0.8660
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /search HTTP/1.1" 200 - 0.8680
2012-01-31T21:55:01.387000Z file://speedy//var/log/cron.log: Jan 31 13:55:01 speedy /USR/SBIN/CRON[26208]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2012-01-31T21:55:01.401000Z file://speedy//var/log/auth.log: Jan 31 13:55:01 speedy CRON[26207]: pam_unix(cron:session): session opened for user root by (uid=0)
2012-01-31T21:55:01.402000Z file://speedy//var/log/auth.log: Jan 31 13:55:01 speedy CRON[26207]: pam_unix(cron:session): session closed for user root
2012-01-31T21:55:01.405000Z file://speedy//var/log/syslog: Jan 31 13:55:01 speedy /USR/SBIN/CRON[26208]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /js/jquery-ui-1.8.13.min.js HTTP/1.1" 200 200104 0.2790
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /js/jquery-ui-1.8.13.min.js HTTP/1.1" 200 - 0.2820
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /js/jquery.tmpl.min.js HTTP/1.1" 200 6007 0.4010
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /js/jquery.tmpl.min.js HTTP/1.1" 200 - 0.4030
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /css/smoothness/jquery-ui-1.8.5.custom.css HTTP/1.1" 200 33959 0.3950
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /css/smoothness/jquery-ui-1.8.5.custom.css HTTP/1.1" 200 - 0.3960
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /js/jquery-1.6.1.min.js HTTP/1.1" 200 91342 0.4620
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /js/jquery-1.6.1.min.js HTTP/1.1" 200 - 0.4620
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /js/flot/jquery.flot.js HTTP/1.1" 200 89666 0.2480
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /js/flot/jquery.flot.js HTTP/1.1" 200 - 0.2490
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /js/jquery-hashchange-1.0.0.js HTTP/1.1" 200 3118 0.3780
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /js/jquery-hashchange-1.0.0.js HTTP/1.1" 200 - 0.3790
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /js/logstash.js HTTP/1.1" 200 12819 0.2810
127.0.0.1 - - [31/Jan/2012 13:55:01] "GET /js/logstash.js HTTP/1.1" 200 - 0.2830
127.0.0.1 - - [31/Jan/2012 13:55:02] "GET /style.css HTTP/1.1" 200 1385 1.1970
127.0.0.1 - - [31/Jan/2012 13:55:02] "GET /style.css HTTP/1.1" 200 - 1.2020
NativeException - org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [initial], No indices / shards to search on, requested indices are []:
org/elasticsearch/action/search/type/TransportSearchTypeAction.java:128:in `<init>'
org/elasticsearch/action/search/type/TransportSearchQueryThenFetchAction.java:72:in `<init>'
org/elasticsearch/action/search/type/TransportSearchQueryThenFetchAction.java:63:in `<init>'
org/elasticsearch/action/search/type/TransportSearchQueryThenFetchAction.java:60:in `doExecute'
org/elasticsearch/action/search/type/TransportSearchQueryThenFetchAction.java:52:in `doExecute'
org/elasticsearch/action/support/BaseAction.java:61:in `execute'
org/elasticsearch/action/search/TransportSearchAction.java:112:in `doExecute'
org/elasticsearch/action/search/TransportSearchAction.java:49:in `doExecute'
org/elasticsearch/action/support/BaseAction.java:61:in `execute'
org/elasticsearch/client/node/NodeClient.java:186:in `search'
org/elasticsearch/client/action/search/SearchRequestBuilder.java:697:in `doExecute'
org/elasticsearch/client/action/support/BaseRequestBuilder.java:56:in `execute'
org/elasticsearch/client/action/support/BaseRequestBuilder.java:51:in `execute'
sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
sun/reflect/NativeMethodAccessorImpl.java:57:in `invoke'
sun/reflect/DelegatingMethodAccessorImpl.java:43:in `invoke'
java/lang/reflect/Method.java:616:in `invoke'
org/jruby/javasupport/JavaMethod.java:508:in `invokeDirectWithExceptionHandling'
org/jruby/javasupport/JavaMethod.java:368:in `invokeDirect'
org/jruby/java/invokers/InstanceMethodInvoker.java:50:in `call'
org/jruby/runtime/callsite/CachingCallSite.java:292:in `cacheAndCall'
org/jruby/runtime/callsite/CachingCallSite.java:135:in `call'
org/jruby/ast/CallNoArgNode.java:63:in `interpret'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/jruby-elasticsearch-0.0.11/lib/jruby-elasticsearch/searchrequest.rb:52:in `execute!'
file:/tmp/logstash-1.1.0-monolithic.jar!/logstash/search/elasticsearch.rb:186:in `histogram'
file:/tmp/logstash-1.1.0-monolithic.jar!/logstash/web/controllers/api_v1.rb:160:in `HEAD /api/histogram'
org/jruby/RubyMethod.java:117:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:1212:in `compile!'
org/jruby/RubyProc.java:258:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:772:in `route!'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:788:in `route_eval'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:772:in `route!'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:821:in `process_route'
org/jruby/RubyKernel.java:1206:in `catch'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:819:in `process_route'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:771:in `route!'
org/jruby/RubyArray.java:1612:in `each'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:770:in `route!'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:886:in `dispatch!'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:706:in `call!'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:871:in `invoke'
org/jruby/RubyKernel.java:1206:in `catch'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:871:in `invoke'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:706:in `call!'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:692:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-1.3.4/lib/rack/showexceptions.rb:24:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-1.3.4/lib/rack/commonlogger.rb:20:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-protection-1.1.4/lib/rack/protection/xss_header.rb:22:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-protection-1.1.4/lib/rack/protection/path_traversal.rb:16:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-protection-1.1.4/lib/rack/protection/json_csrf.rb:17:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-protection-1.1.4/lib/rack/protection/base.rb:47:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-protection-1.1.4/lib/rack/protection/xss_header.rb:22:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-1.3.4/lib/rack/logger.rb:15:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-1.3.4/lib/rack/commonlogger.rb:20:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-1.3.4/lib/rack/head.rb:9:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/showexceptions.rb:21:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/mizuno-0.5.0/lib/mizuno/rack_servlet.rb:75:in `service'
org/jruby/RubyKernel.java:1206:in `catch'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/mizuno-0.5.0/lib/mizuno/rack_servlet.rb:74:in `service'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/mizuno-0.5.0/lib/mizuno/rack_servlet.rb:236:in `handle_exceptions'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/mizuno-0.5.0/lib/mizuno/rack_servlet.rb:45:in `service'NativeException: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [initial], No indices / shards to search on, requested indices are []
org/elasticsearch/action/search/type/TransportSearchTypeAction.java:128:in `<init>'
org/elasticsearch/action/search/type/TransportSearchQueryThenFetchAction.java:72:in `<init>'
org/elasticsearch/action/search/type/TransportSearchQueryThenFetchAction.java:63:in `<init>'
org/elasticsearch/action/search/type/TransportSearchQueryThenFetchAction.java:60:in `doExecute'
org/elasticsearch/action/search/type/TransportSearchQueryThenFetchAction.java:52:in `doExecute'
org/elasticsearch/action/support/BaseAction.java:61:in `execute'
org/elasticsearch/action/search/TransportSearchAction.java:112:in `doExecute'
org/elasticsearch/action/search/TransportSearchAction.java:49:in `doExecute'
org/elasticsearch/action/support/BaseAction.java:61:in `execute'
org/elasticsearch/client/node/NodeClient.java:186:in `search'
org/elasticsearch/client/action/search/SearchRequestBuilder.java:697:in `doExecute'
org/elasticsearch/client/action/support/BaseRequestBuilder.java:56:in `execute'
org/elasticsearch/client/action/support/BaseRequestBuilder.java:51:in `execute'
sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
sun/reflect/NativeMethodAccessorImpl.java:57:in `invoke'
sun/reflect/DelegatingMethodAccessorImpl.java:43:in `invoke'
java/lang/reflect/Method.java:616:in `invoke'
org/jruby/javasupport/JavaMethod.java:508:in `invokeDirectWithExceptionHandling'
org/jruby/javasupport/JavaMethod.java:368:in `invokeDirect'
org/jruby/java/invokers/InstanceMethodInvoker.java:50:in `call'
org/jruby/runtime/callsite/CachingCallSite.java:292:in `cacheAndCall'
org/jruby/runtime/callsite/CachingCallSite.java:135:in `call'
org/jruby/ast/CallNoArgNode.java:63:in `interpret'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/jruby-elasticsearch-0.0.11/lib/jruby-elasticsearch/searchrequest.rb:52:in `execute!'
file:/tmp/logstash-1.1.0-monolithic.jar!/logstash/search/elasticsearch.rb:186:in `histogram'
file:/tmp/logstash-1.1.0-monolithic.jar!/logstash/web/controllers/api_v1.rb:160:in `HEAD /api/histogram'
org/jruby/RubyMethod.java:117:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:1212:in `compile!'
org/jruby/RubyProc.java:258:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:772:in `route!'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:788:in `route_eval'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:772:in `route!'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:821:in `process_route'
org/jruby/RubyKernel.java:1206:in `catch'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:819:in `process_route'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:771:in `route!'
org/jruby/RubyArray.java:1612:in `each'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:770:in `route!'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:886:in `dispatch!'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:706:in `call!'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:871:in `invoke'
org/jruby/RubyKernel.java:1206:in `catch'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:871:in `invoke'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:706:in `call!'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/base.rb:692:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-1.3.4/lib/rack/showexceptions.rb:24:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-1.3.4/lib/rack/commonlogger.rb:20:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-protection-1.1.4/lib/rack/protection/xss_header.rb:22:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-protection-1.1.4/lib/rack/protection/path_traversal.rb:16:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-protection-1.1.4/lib/rack/protection/json_csrf.rb:17:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-protection-1.1.4/lib/rack/protection/base.rb:47:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-protection-1.1.4/lib/rack/protection/xss_header.rb:22:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-1.3.4/lib/rack/logger.rb:15:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-1.3.4/lib/rack/commonlogger.rb:20:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/rack-1.3.4/lib/rack/head.rb:9:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/sinatra-1.3.1/lib/sinatra/showexceptions.rb:21:in `call'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/mizuno-0.5.0/lib/mizuno/rack_servlet.rb:75:in `service'
org/jruby/RubyKernel.java:1206:in `catch'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/mizuno-0.5.0/lib/mizuno/rack_servlet.rb:74:in `service'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/mizuno-0.5.0/lib/mizuno/rack_servlet.rb:236:in `handle_exceptions'
file:/tmp/logstash-1.1.0-monolithic.jar!/gems/mizuno-0.5.0/lib/mizuno/rack_servlet.rb:45:in `service'127.0.0.1 - - [31/Jan/2012 13:55:02] "GET /api/histogram?offset=0&count=50&q=*+%40timestamp%3A%5B2012-01-24+TO+2012-02-02%5D&interval=3600000 HTTP/1.1" 500 5390 0.1570
127.0.0.1 - - [31/Jan/2012 13:55:02] "GET /api/histogram?offset=0&count=50&q=*+%40timestamp%3A%5B2012-01-24+TO+2012-02-02%5D&interval=3600000 HTTP/1.1" 500 - 0.1590
127.0.0.1 - - [31/Jan/2012 13:55:02] "GET /media/throbber.gif HTTP/1.1" 200 2608 0.2190
127.0.0.1 - - [31/Jan/2012 13:55:02] "GET /media/throbber.gif HTTP/1.1" 200 - 0.2320
127.0.0.1 - - [31/Jan/2012 13:55:02] "POST /api/search?format=html HTTP/1.1" 500 287 0.3120
127.0.0.1 - - [31/Jan/2012 13:55:02] "POST /api/search?format=html HTTP/1.1" 500 - 0.3130
127.0.0.1 - - [31/Jan/2012 13:55:02] "GET /favicon.ico HTTP/1.1" 200 13 0.0290
127.0.0.1 - - [31/Jan/2012 13:55:02] "GET /favicon.ico HTTP/1.1" 200 - 0.0310
Raw
mylogstash.conf
input {
file {
type => "linux-syslog"
# Wildcards work, here :)
path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]
}
file {
type => "apache-access"
path => "/var/log/apache2/access.log"
}
file {
type => "apache-error"
path => "/var/log/apache2/error.log"
}
}
output {
# Emit events to stdout for easy debugging of what is going through
# logstash.
stdout { }
# This will use elasticsearch to store your logs.
# The 'embedded' option will cause logstash to run the elasticsearch
# server in the same process, so you don't have to worry about
# how to download, configure, or run elasticsearch!
elasticsearch { embedded => true }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment