td-agentのs3 Pluginを使っている

td-agent, s3 plugin

$curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh
$sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-parser
$sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-s3
$sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-bigquery
$sudo chmod 755 -R  /var/log/httpd/

$sudo vi /etc/td-agent/td-agent.conf

###
## Output descriptions:
##

<match apache.access>
  type s3

  aws_key_id xxxxxxxxxxxxxxxxxxxx
  aws_sec_key xxxxxxxxxxxxxxxxxxxxxx
  s3_bucket bucket-td-access
  s3_region ap-northeast-1
  s3_object_key_format %{path}%{time_slice}_%{index}_%{hostname}.%{file_extension}
  path access_logs/
  buffer_path /var/log/td-agent/buffer

  flush_interval 60s
  time_slice_format %Y%m%d/%Y%m%d-%H
</match>

###
## Source descriptions:
###

<source>
        type tail
        format ltsv
        path /path/
        tag apache.access
        pos_file /tmp/td-agent/apache.pos
</source>






参考サイト

Installing Fluentd Using rpm Package
http://docs.fluentd.org/articles/install-by-rpm

EC2のログをfluentdを使ってタグ情報を利用したフォルダ構成でS3にアップロードする
http://qiita.com/toshihirock/items/3e59dc20998981647d8c

Fluentdで、RailsのログをS3に保存する方法
http://qiita.com/jwako/items/22394a9755ed64504161

fluent-plugin-forestが見つからない！
http://qiita.com/pman-taichi/items/24d05f40c8d292e6698c

fluentdのoutput_s3プラグイン
http://qiita.com/munazo/items/d665e2c4e86f42d06cf7

awsのs3を操作する為のaccess keyとsecret keyを取得する(IAM)
http://joppot.info/2014/06/14/1621

ltsvがいいのか。

https://codeiq.jp/magazine/2014/08/12686/