Skip to content

Instantly share code, notes, and snippets.

@twingo-b
Created June 10, 2014 00:41
Show Gist options
  • Select an option

  • Save twingo-b/0a0fb59bdc03a6e641c9 to your computer and use it in GitHub Desktop.

Select an option

Save twingo-b/0a0fb59bdc03a6e641c9 to your computer and use it in GitHub Desktop.
fluent-plugin-bigquery利用時に、tableを動的に設定する ref: http://qiita.com/twingo-b@github/items/864a3eb22083e7f0f239
<source>
type tail
format apache
time_format %d/%b/%Y:%H:%M:%S %z
path /var/log/httpd/access_log
tag td.apache_access_log
pos_file /var/lib/fluent/td.apache_access_log.pos
</source>
...
<match td.apache_access_log>
type record_reformer
enable_ruby true
tag ${tag}.${time.strftime('%Y%m')}
<record>
input_tag ${tag}
port ${port}
host ${host}
user ${user}
method ${method}
path ${path}
http_version ${http_version}
conde ${conde}
size ${size}
referer ${referer}
agent ${agent}
</record>
</match>
<match td.**>
type forest
subtype copy
remove_prefix td
<template>
# Amazon S3
<store>
type s3
s3_bucket s3-bucket
path /
buffer_path /var/log/td-agent/buffer/s3.${tag}.*.buffer
time_slice_format ${tag_parts[0]}/%Y/%Y%m%d/${tag}-%Y%m%d-%H
</store>
</template>
# Google BigQuery
<case apache_access_log.**>
<store>
type bigquery
method insert
auth_method private_key
email XXXXXXXXXXXX@developer.gserviceaccount.com
private_key_path /etc/td-agent/XXXXXXXXXXXX-privatekey.p12
project your-project-id
dataset your_dataset
table apache_access_log_${tag_parts[-1]}
time_format %s
time_field time
schema_path /etc/td-agent/apache_access_log_schema.json
</store>
</case>
</match>
but, please note that enabling ruby codes is not encouraged by security reasons and also in terms of the performance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment