Skip to content

Instantly share code, notes, and snippets.

@masayuki5160

masayuki5160/google-big-queryの場合のconf

Last active June 3, 2016 10:23
Show Gist options
  • Save masayuki5160/4b332bfd9151241730ef to your computer and use it in GitHub Desktop.
Save masayuki5160/4b332bfd9151241730ef to your computer and use it in GitHub Desktop.
Download ZIP
fluentdの設定をいろいろ試して検証している
Raw
google-big-queryの場合のconf
<source>
type forward
</source>
## Processor Server
<match apache.access>
type bigquery
method insert
auth_method private_key
email [email protected]
private_key_path ./xxxxxxx.p12
#private_key_passphrase notasecret # default
project hoge-pj
dataset hoge_dataset
table hogehoge_table
time_format %s
time_field time
schema_path /etc/td-agent/schema.json
</match>
Raw
ログの整形(Processor)するconf
<source>
type forward
</source>
## To S3(この中でログを整形するとか？)
<match apache.access>
type s3
aws_key_id xxxxxxxxx
aws_sec_key xxxxxxxxx
s3_bucket masayuki-td-test
s3_region ap-northeast-1
s3_object_key_format %{path}%{time_slice}_%{index}_%{hostname}.%{file_extension}
path access_logs_from_processor/
buffer_path /var/log/td-agent/buffer
flush_interval 60s
time_slice_format %Y%m%d/%Y%m%d-%H
</match>
Raw
ログ送信元サーバのconf
<source>
type tail
format apache
path /var/log/httpd/access_log
tag apache.access
pos_file /tmp/td-agent/apache.pos
</source>
## To Aggregate server
<match apache.access>
type forward
<server>
host [host-name or IP]
port 24224
</server>
</match>
Raw
ログ集約サーバのconf
<source>
type forward
</source>
## Processor Server
<match apache.access>
type forward
<server>
host [host-name/IP]
port 24224
</server>
</match>
## S3
<match apache.access>
type s3
aws_key_id xxxxxxx
aws_sec_key xxxxxxx
s3_bucket masayuki-td-test
s3_region ap-northeast-1
s3_object_key_format %{path}%{time_slice}_%{index}_%{hostname}.%{file_extension}
path access_logs/
buffer_path /var/log/td-agent/buffer
flush_interval 60s
time_slice_format %Y%m%d/%Y%m%d-%H
</match>
@masayuki5160
Copy link
Author

プラグインはすっとはいる。

$ sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-bigquery

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment