Skip to content

Instantly share code, notes, and snippets.

@matsumotory

matsumotory/td-agent.con

Created October 30, 2015 02:07
Show Gist options
  • Save matsumotory/901ab204076c9cbc947b to your computer and use it in GitHub Desktop.
Save matsumotory/901ab204076c9cbc947b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
td-agent.con
# from mod_resource_checker log
<source>
type tail
path /var/log/httpd/resource.log
pos_file /var/log/td-agent/resource.log.pos
tag apache.resource
format json
</source>
<filter apache.resource.**>
type record_transformer
<record>
real_hostname ${hostname}
</record>
</filter>
# into Norikra
<match apache.resource.**>
type norikra
norikra 127.0.0.1:26571
remove_tag_prefix apache
target_map_tag true
</match>
# from Norikra
<source>
type norikra
norikra 127.0.0.1:26571
<fetch>
method sweep
tag_prefix norikra.query
tag query_name
interval 1m
</fetch>
</source>
# for status code
<match norikra.query.status.**>
type json_expander
subtype growthforecast
remove_prefix norikra.query.status
<template>
gfapi_url http://127.0.0.1:5125/api/
graph_path ${data[real_hostname]}/${data[hostname]}/${key_name}
name_keys count_2xx,count_3xx,count_4xx,count_5xx
</template>
</match>
# for status code
<match norikra.query.resources.**>
type json_expander
subtype growthforecast
remove_prefix norikra.query.resources
<template>
gfapi_url http://127.0.0.1:5125/api/
graph_path ${data[real_hostname]}/${data[hostname]}/${key_name}
name_keys ucpu,scpu,memory,total_size,total_content_length
</template>
</match>
Raw
td-agent.log
2015-10-29 20:19:29 +0900 [info]: starting fluentd-0.12.12
2015-10-29 20:19:29 +0900 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-10-29 20:19:29 +0900 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-10-29 20:19:29 +0900 [info]: gem 'fluent-plugin-forest' version '0.3.0'
2015-10-29 20:19:29 +0900 [info]: gem 'fluent-plugin-growthforecast' version '0.2.9'
2015-10-29 20:19:29 +0900 [info]: gem 'fluent-plugin-json_expander' version '0.0.2'
2015-10-29 20:19:29 +0900 [info]: gem 'fluent-plugin-mongo' version '0.7.10'
2015-10-29 20:19:29 +0900 [info]: gem 'fluent-plugin-norikra' version '0.3.0'
2015-10-29 20:19:29 +0900 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2015-10-29 20:19:29 +0900 [info]: gem 'fluent-plugin-s3' version '0.5.9'
2015-10-29 20:19:29 +0900 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2015-10-29 20:19:29 +0900 [info]: gem 'fluent-plugin-td' version '0.10.27'
2015-10-29 20:19:29 +0900 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.1'
2015-10-29 20:19:29 +0900 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2015-10-29 20:19:29 +0900 [info]: gem 'fluentd' version '0.12.12'
2015-10-29 20:19:29 +0900 [info]: adding filter pattern="apache.resource.**" type="record_transformer"
2015-10-29 20:19:29 +0900 [info]: adding match pattern="apache.resource.**" type="norikra"
2015-10-29 20:19:29 +0900 [info]: adding match pattern="norikra.query.status.**" type="json_expander"
2015-10-29 20:19:29 +0900 [info]: adding match pattern="norikra.query.resources.**" type="json_expander"
2015-10-29 20:19:29 +0900 [info]: adding source type="tail"
2015-10-29 20:19:29 +0900 [info]: adding source type="norikra"
2015-10-29 20:19:29 +0900 [info]: using configuration file: <ROOT>
<source>
type tail
path /var/log/httpd/resource.log
pos_file /var/log/td-agent/resource.log.pos
tag apache.resource
format json
</source>
<filter apache.resource.**>
type record_transformer
<record>
real_hostname ${hostname}
</record>
</filter>
<match apache.resource.**>
type norikra
norikra 127.0.0.1:26571
remove_tag_prefix apache
target_map_tag true
</match>
<source>
type norikra
norikra 127.0.0.1:26571
<fetch>
method sweep
tag_prefix norikra.query
tag query_name
interval 1m
</fetch>
</source>
<match norikra.query.status.**>
type json_expander
subtype growthforecast
remove_prefix norikra.query.status
<template>
gfapi_url http://127.0.0.1:5125/api/
graph_path ${data[real_hostname]}/${data[hostname]}/${key_name}
name_keys count_2xx,count_3xx,count_4xx,count_5xx
</template>
</match>
<match norikra.query.resources.**>
type json_expander
subtype growthforecast
remove_prefix norikra.query.resources
<template>
gfapi_url http://127.0.0.1:5125/api/
graph_path ${data[real_hostname]}/${data[hostname]}/${key_name}
name_keys ucpu,scpu,memory,total_size,total_content_length
</template>
</match>
</ROOT>
2015-10-29 20:19:29 +0900 [warn]: parameter 'gfapi_url' in <template>
gfapi_url http://127.0.0.1:5125/api/
graph_path ${data[real_hostname]}/${data[hostname]}/${key_name}
name_keys count_2xx,count_3xx,count_4xx,count_5xx
</template> is not used.
2015-10-29 20:19:29 +0900 [warn]: parameter 'graph_path' in <template>
gfapi_url http://127.0.0.1:5125/api/
graph_path ${data[real_hostname]}/${data[hostname]}/${key_name}
name_keys count_2xx,count_3xx,count_4xx,count_5xx
</template> is not used.
2015-10-29 20:19:29 +0900 [warn]: parameter 'name_keys' in <template>
gfapi_url http://127.0.0.1:5125/api/
graph_path ${data[real_hostname]}/${data[hostname]}/${key_name}
name_keys count_2xx,count_3xx,count_4xx,count_5xx
</template> is not used.
2015-10-29 20:19:29 +0900 [warn]: parameter 'graph_path' in <template>
gfapi_url http://127.0.0.1:5125/api/
graph_path ${data[real_hostname]}/${data[hostname]}/${key_name}
name_keys count_2xx,count_3xx,count_4xx,count_5xx
</template> is not used.
2015-10-29 20:19:29 +0900 [warn]: parameter 'name_keys' in <template>
gfapi_url http://127.0.0.1:5125/api/
graph_path ${data[real_hostname]}/${data[hostname]}/${key_name}
name_keys count_2xx,count_3xx,count_4xx,count_5xx
</template> is not used.
2015-10-29 20:19:29 +0900 [warn]: parameter 'gfapi_url' in <template>
gfapi_url http://127.0.0.1:5125/api/
graph_path ${data[real_hostname]}/${data[hostname]}/${key_name}
name_keys ucpu,scpu,memory,total_size,total_content_length
</template> is not used.
2015-10-29 20:19:29 +0900 [warn]: parameter 'graph_path' in <template>
gfapi_url http://127.0.0.1:5125/api/
graph_path ${data[real_hostname]}/${data[hostname]}/${key_name}
name_keys ucpu,scpu,memory,total_size,total_content_length
</template> is not used.
2015-10-29 20:19:29 +0900 [warn]: parameter 'name_keys' in <template>
gfapi_url http://127.0.0.1:5125/api/
graph_path ${data[real_hostname]}/${data[hostname]}/${key_name}
name_keys ucpu,scpu,memory,total_size,total_content_length
</template> is not used.
2015-10-29 20:19:29 +0900 [info]: following tail of /var/log/httpd/resource.log
2015-10-29 20:19:30 +0900 [info]: [out_json_expand] Expanded new output: growthforecast
2015-10-29 20:19:30 +0900 [warn]: no patterns matched tag="norikra.query.response_time"
2015-10-29 20:19:30 +0900 [warn]: no patterns matched tag="norikra.query.response_time"
2015-10-29 20:19:30 +0900 [warn]: no patterns matched tag="norikra.query.response_time"
2015-10-29 20:19:30 +0900 [info]: [out_json_expand] Expanded new output: growthforecast
2015-10-29 20:20:31 +0900 [info]: [out_json_expand] Expanded new output: growthforecast
2015-10-29 20:20:31 +0900 [info]: [out_json_expand] Expanded new output: growthforecast
2015-10-29 20:22:32 +0900 [info]: [out_json_expand] Expanded new output: growthforecast
2015-10-29 20:22:32 +0900 [info]: [out_json_expand] Expanded new output: growthforecast
2015-10-30 05:07:02 +0900 [info]: force flushing buffered events
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment