Skip to content

Instantly share code, notes, and snippets.

@kiyoto

kiyoto/update_flume_plugin.markdown

Last active August 29, 2015 14:08
Show Gist options
  • Save kiyoto/2b096c3d293a6875e9c6 to your computer and use it in GitHub Desktop.
Save kiyoto/2b096c3d293a6875e9c6 to your computer and use it in GitHub Desktop.
Download ZIP
Going about updateing fluent-plugin-flume
Raw
update_flume_plugin.markdown

This is a little note about how I would go about updating fluent-plugin-flume to resolve this issue.

Cause

The current fluent-plugin-flume supports 0.9 protocol. However, most curernt deployments are on the 1.5 branch. This causes a compatibility issue because the Thrift definition for Flume changed between 0.9 and 1.5.

How to fix it

  1. We need to update the flume.thrift file. The Thrift file definition for Flume 1.5 can be found here.
  2. The actual Thrift server code in Ruby should be generated using Thrift compiler. The 0.9.1 release should work. Use the updated flume.thrift to generate the server code (in lib/fluent/plugin/thrift)
  3. Probably it is best to keep support flume-og. One way to do it is to keep the current Thrift code, probably in a different subdirectory, and switch the "require" statement based on a configuration parameter. This is what the Flumelogger project does: https://github.com/lsjostro/ruby-flumelogger/blob/master/lib/flume-logger/logger.rb#L59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment