kiyoto

-------------------------------------------------------------------
test_in_exec.rb
-------------------------------------------------------------------
Run options: 

# Running tests:

......

Finished tests in 8.276824s, 0.7249 tests/s, 1.8123 assertions/s.

kiyoto

<source>
  type tail
  tag apache.access
  format apache2
  path /path/to/your/log
  read_from_head true #過去ログ用。ファイルの先頭から読む
</source>

<match apache.access>
  type elasticsearch

kiyoto

With this config, if your hostname is koala and if you have files like

log/t1.log
   /t2.log
   /...

They are sent to

kiyoto

require 'fluent-logger'

# specify the host and port to forward data to
logger = Fluent::Logger.new(nil, :host => "localhost", :port => 24224)

fn = ARGV[0]
fh = File.new(fn)

msgpack_unpacker = MessagePack::Unpacker.new(fh)

kiyoto

module Fluent
  class IfFilter < Filter

    Fluent::Plugin.register_filter('if', self)

    config_param :if, :string
    def configure(conf)
      super
      @placeholder_expander = RubyPlaceholderExpander.new(log)
      

kiyoto

<source>
  type tail
  format json
  path /path/to/file
  tag  aoi.yu
</source>

kiyoto

Versioning Fluentd's Documentation

As Fluentd starts rolling out new major uprates (v0.12 & v0.14) and inches towards the v1 release, I want to re-organize and version the docs website

Here is my plan.

1. All current URLs will start pointing to the latest release. This does NOT include preview releases.
2. For each page, the version information is embedded as follows: https://docs.fluentd.org/<LANGUAGE>/<VERSION>/<PAGE_NAME>. Ex. https://docs.fluentd.org/ja/v0-12/config-file. The pages that have version info in the URL are called Versioned Pages.
3. For each Versioned Page that is not the latest version, the following warning shows up on the page:

kiyoto

require 'nokogiri'
require 'open-uri'
require 'json'

BASE_URL = 'http://www.tomtunguz.com'

class URLFormatError < RuntimeError; end

def fetch_categories(base_url)
	html = open(base_url).read

kiyoto

This is a little note about how I would go about updating fluent-plugin-flume to resolve this issue.

Cause

The current fluent-plugin-flume supports 0.9 protocol. However, most curernt deployments are on the 1.5 branch. This causes a compatibility issue because the Thrift definition for Flume changed between 0.9 and 1.5.

How to fix it

1. We need to update the flume.thrift file. The Thrift file definition for Flume 1.5 can be found here.
2. The actual Thrift server code in Ruby should be generated using Thrift compiler. The 0.9.1 release should work. Use the updated flume.thrift to generate the server code (in lib/fluent/plugin/thrift)

kiyoto

# sender td-agent.conf
#---------------------------------------------------

<source>
  type forward
</source>

<match apache.*>
   type secure_forward
   shared_key somekey