Created
June 10, 2014 00:41
-
-
Save twingo-b/0a0fb59bdc03a6e641c9 to your computer and use it in GitHub Desktop.
fluent-plugin-bigquery利用時に、tableを動的に設定する ref: http://qiita.com/twingo-b@github/items/864a3eb22083e7f0f239
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <source> | |
| type tail | |
| format apache | |
| time_format %d/%b/%Y:%H:%M:%S %z | |
| path /var/log/httpd/access_log | |
| tag td.apache_access_log | |
| pos_file /var/lib/fluent/td.apache_access_log.pos | |
| </source> | |
| ... | |
| <match td.apache_access_log> | |
| type record_reformer | |
| enable_ruby true | |
| tag ${tag}.${time.strftime('%Y%m')} | |
| <record> | |
| input_tag ${tag} | |
| port ${port} | |
| host ${host} | |
| user ${user} | |
| method ${method} | |
| path ${path} | |
| http_version ${http_version} | |
| conde ${conde} | |
| size ${size} | |
| referer ${referer} | |
| agent ${agent} | |
| </record> | |
| </match> | |
| <match td.**> | |
| type forest | |
| subtype copy | |
| remove_prefix td | |
| <template> | |
| # Amazon S3 | |
| <store> | |
| type s3 | |
| s3_bucket s3-bucket | |
| path / | |
| buffer_path /var/log/td-agent/buffer/s3.${tag}.*.buffer | |
| time_slice_format ${tag_parts[0]}/%Y/%Y%m%d/${tag}-%Y%m%d-%H | |
| </store> | |
| </template> | |
| # Google BigQuery | |
| <case apache_access_log.**> | |
| <store> | |
| type bigquery | |
| method insert | |
| auth_method private_key | |
| email XXXXXXXXXXXX@developer.gserviceaccount.com | |
| private_key_path /etc/td-agent/XXXXXXXXXXXX-privatekey.p12 | |
| project your-project-id | |
| dataset your_dataset | |
| table apache_access_log_${tag_parts[-1]} | |
| time_format %s | |
| time_field time | |
| schema_path /etc/td-agent/apache_access_log_schema.json | |
| </store> | |
| </case> | |
| </match> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| but, please note that enabling ruby codes is not encouraged by security reasons and also in terms of the performance. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment