#FluentdのFormat追加の名前について
##名前に関する規則
個人的にはext_format_<format name>という感じがいいと思っています。<format name>に関しては、全部一か所に突っ込んでしまうと、場合によってはいらないパーサーもロードされるので、ある程度グルーピングするとよいかなと。例えばMTA系ののフォーマットに関するものはext_format_mailとか、MongoDBに関するフォーマットはext_format_mongoとか。でもって必要なext_format_<format name>をPluginディレクトリに置く・gem installするという感じ。
このアプローチの最大の問題は、ext_format_*系が量産される可能性があること。これに関しては、ある程度使われるとわかったものは、随時Coreにマージしてもよいかもしれない。
##フォルダ構成
| module Fluent | |
| class HerokuHttpInput < Input | |
| Plugin.register_input('heroku_http', self) | |
| include DetachMultiProcessMixin | |
| require 'http/parser' | |
| def initialize | |
| require 'webrick/httputils' |
One of the common questions on Fluentd is "does Fluentd handle my log?" At the moment, the answer to this question comes down to one of the following three:
While we can't expect to meet all needs with our core plugins (and that's why there are 3rd party plugins), we want to make things easier and simpler.
One idea that I've been thinking about is adding Grok-like parser. For those who don't know what Grok is, it's a regex macro library originally developed by Jordan Sissel (of Logstash). A big upshot here is that a lot of grok patterns have already been written, and we can immediately take advantage of them.
| FROM base | |
| MAINTAINER repeatedly "https://github.com/repeatedly" | |
| RUN apt-get install -y --force-yes curl | |
| RUN curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-precise.sh | sh | |
| RUN sed -i 's/ulimit/# ulimit/g' /etc/init.d/td-agent | |
| RUN echo "ulimit -n 8192" >> /etc/default/td-agent |
| <source> | |
| type tail | |
| format apache2 | |
| path /var/log/apache2/access_log | |
| pos_file /var/log/fluentd/apache2.access_log.pos | |
| tag riak.apache | |
| </source> | |
| <match riak.**> | |
| type riak |
| 2013-3-03 14:27:33 [main] INFO Main - Start | |
| 2013-3-03 14:27:33 [main] ERROR Main - Exception | |
| javax.management.RuntimeErrorException: null | |
| at Main.main(Main.java:16) ~[bin/:na] | |
| 2013-3-03 14:27:33 [main] INFO Main - End |
| 2.0.0-p0 :006 > g = Grok.new | |
| => #<Grok:0x007f9b5aa85988> | |
| 2.0.0-p0 :007 > g.discover("hello") | |
| NameError: uninitialized constant Grok::GrokDiscover | |
| from /Users/owenestea/.rvm/gems/ruby-2.0.0-p0/gems/jls-grok-0.10.12/lib/grok.rb:127:in `init_discover' | |
| from /Users/owenestea/.rvm/gems/ruby-2.0.0-p0/gems/jls-grok-0.10.12/lib/grok.rb:120:in `discover' | |
| from (irb):7 | |
| from /Users/owenestea/.rvm/rubies/ruby-2.0.0-p0/bin/irb:16:in `<main>' |
| <source> | |
| type embedded_elasticsearch | |
| </source> | |
| <source> | |
| type kibana_server | |
| bind 0.0.0.0 | |
| port 24300 | |
| mount /kibana/ | |
| access_log_path var/log/kibana/access.log |
| C:/Ruby200-x64/bin/ruby.exe extconf.rb | |
| checking for rb_thread_blocking_region()... yes | |
| checking for rb_thread_alone()... yes | |
| checking for rb_str_set_len()... yes | |
| checking for clock_gettime() in -lrt... no | |
| checking for sys/select.h... no | |
| checking for poll.h... no | |
| checking for sys/epoll.h... no | |
| checking for sys/event.h... no | |
| checking for port.h... no |
| # Twitter input: See https://github.com/y-ken/fluent-plugin-twitter | |
| <source> | |
| type twitter | |
| consumer_key YOU_CONSUTMER_KEY # Required | |
| consumer_secret YOUR_CONSUMER_SECRET # Required | |
| oauth_token YOUR_OAUTH_TOKEN # Required | |
| oauth_token_secret YOUR_TOKEN_SECRET # Required | |
| tag twitter.timeline # Required | |
| timeline userstream # Required s(sampling or userstream) | |
| #keyword SFGiants |