README.md

With this config, if your hostname is koala and if you have files like

log/t1.log
   /t2.log
   /...

They are sent to

receiver/koala.t1.20141219.log
        /koala.t2.20141219.log
        /...

Right now, the time_format must appear in out_file. If you don't want it in the file name, you can embedded in the path by replacing

path receiver/${tag_parts[1..-2]}.*.log

with

path receiver/*/${tag_parts[1..-2]}.log

in receiver.conf. That would change the file structure to

receiver/20141219/koala.t1.log
        /20141219/koala.t2.log
        /...

receiver.conf

<source>
  type forward
  port 24224
</source>

<match with_hostname.**>
  type forest
  subtype file
  <template>
    path receiver/${tag_parts[1..-2]}.*.log
    append true
    flush_interval 1s
  </template>
</match>

sender.conf

<source>
  type tail
  format none
  path log/*.log
  read_from_head true
  tag no_hostname.*
</source>

<match no_hostname.**>
  type record_reformer
  tag  with_hostname.${hostname}.${tag_suffix[-2]}
</match>

<match with_hostname.**>
  type forward
  flush_interval 1s
  <server>
    host localhost
    port 24224
  </server>
</match>

You need to install fluent-plugin-forest and fluent-plugin-record-reformer.

that worked great using :
path receiver/${tag_parts[1..-2]}.*.log

then i decided to change the output path to receiver/hostname/date/original_logfilename.log using :-
path receiver/${tag_parts[1]}/*/${tag_parts[-2]}.log

2014-12-19 22:54:40 +1100 [info]: plugin/out_forest.rb:138:plant: out_forest plants new output: file for tag 'with_hostname.clienthost.wifi.log'
2014-12-19 22:54:40 +1100 [warn]: fluent/engine.rb:163:rescue in emit_stream: emit transaction failed error_class=Errno::ENOENT error=#<Errno::ENOENT: No such file or directory @ rb_sysopen - receiver/clienthost/20141219.b50a905fb6d385fd9/wifi.log>

i then tried your suggestion of :-
path receiver/*/${tag_parts[1..-2]}.log
and got a similar result.... "No such file or directory "

any thoughts on whats going on here ? If I can get that path i tried to work, it is my ideal result.

ok looks like i had cut/paste fail.... her eis the full error i was seeing :-

2014-12-19 22:54:40 +1100 [info]: plugin/out_forest.rb:138:plant: out_forest plants new output: file for tag 'with_hostname.clienthost.wifi.log'
2014-12-19 22:54:40 +1100 [warn]: fluent/engine.rb:163:rescue in emit_stream: emit transaction failed error_class=Errno::ENOENT error=#<Errno::ENOENT: No such file or directory @ rb_sysopen - receiver/clienthost/20141219.b50a905fb6d385fd9/wifi.log>

looks like its failing to either create or read from the intermediate buffer file....

aargh, it cut off the error again. i will separate on multiple lines to ensure it works

2014-12-19 22:54:40 +1100 [warn]: fluent/engine.rb:163:rescue in emit_stream: emit transaction failed error_class=Errno::ENOENT error=#
< Errno::ENOENT: No such file or directory @ rb_sysopen - receiver/clienthost/20141219.b50a905fb6d385fd9/wifi.log >

trying to set buffer_path to something higher in path:
<match with_hostname.**>
type forest
subtype file

buffer_path receiver/${tag_parts[1]}
path receiver/${tag_parts[1]}/*/${tag_parts[-2]}.log
append true
flush_interval 1s

this resulted in the log associated with the first message processed in the right place, but messages from other log files produced :-
2014-12-20 14:52:57 +1100 [error]: plugin/out_forest.rb:142:rescue in plant: Cannot output messages with tag 'with_hostname.clienthost.wifi.log'

feels like im on the right path though.

looks like this gets me where i want to be :-
buffer_path receiver/${tag_parts[0]}/${tag_parts[-2]}

going to run tests with more than my single "clienthost" shortly to see how it handles load...
thanks for the assist ;-)

👍