fluentd_takes_swap.md

swapをどんどん食ってしまう状況が発生

環境

• CentOS release 6.4 (Final)
• td-agent-1.1.18-0.x86_64
• fluent-plugin-secure-forward (0.1.6)
• msgpack (0.4.7)

検証構造

[local perl] -24224-> [local fluentd] - secure_forward or forward-> [vm fluentd] -> out_file 

検証方法

100万件ずつ投げて、topでみてCPUが落ち着いたところのmemoryを採取

設定ファイル

• 受け手(vm fluentd)

<source>
  type secure_forward

  shared_key         ${key}
  self_hostname      ${hostname}
  cert_auto_generate     yes
  allow_anonymous_source no
  authentication         yes
  port 24284
  <user>
    username ${user}
    password ${pass}
  </user>

  <client>
    host  ${ip}
  </client>
</source>
<match test>
  type file
  path /tmp/fluentd
  time_slice_format %Y%m%d%H%M
  time_slice_wait 1m
  time_format %Y%m%dT%H%M%S%z
  utc
</match>

• 送り手(local fluentd) (type forwardの時もほぼ同様)

<source>
  type forward
</source>

<match filtered.api>
  type secure_forward
  shared_key ${key}
  self_hostname ${hostname}
  <server>
    host ${vm host}
    port 24284

    username ${user}
    password ${pass}
  </server>
</match>

• スクリプト

perl -MFluent::Logger -e '$logger = Fluent::Logger->new(host=>"localhost", port=>24224); my $cnt = 1_000_000; while($cnt-- > 0){ $logger->post("filtered.api", {name => "from perl"}); }'

流した結果

• secure_forward

タイミング	結果(top)
0	3344 td-agent 20 0 260m 39m 3532 S 0.0 2.1 0:00.32 ruby
1	3344 td-agent 20 0 275m 56m 3568 S 0.3 3.0 0:10.54 ruby
2	3344 td-agent 20 0 303m 82m 3568 S 0.0 4.4 0:21.19 ruby
3	3344 td-agent 20 0 319m 98m 3568 S 0.0 5.3 0:28.91 ruby
4	3344 td-agent 20 0 335m 114m 3568 S 0.0 6.1 0:37.16 ruby
5	3344 td-agent 20 0 367m 144m 3568 S 0.0 7.7 0:49.37 ruby
6	3344 td-agent 20 0 391m 166m 3568 S 0.0 8.9 0:59.89 ruby
7	3344 td-agent 20 0 415m 188m 3568 S 0.3 10.1 1:09.54 ruby
8	3344 td-agent 20 0 435m 209m 3568 S 0.0 11.2 1:17.03 ruby
9	3344 td-agent 20 0 467m 237m 3568 S 0.0 12.7 1:27.86 ruby
10	3344 td-agent 20 0 467m 237m 3568 S 0.0 12.7 1:27.88 ruby

• forward

タイミング	結果(top)
0	3460 td-agent 20 0 229m 31m 2612 S 0.0 1.7 0:00.10 ruby
1	3460 td-agent 20 0 281m 67m 2664 S 0.0 3.6 0:01.97 ruby
2	3460 td-agent 20 0 333m 108m 2664 S 0.0 5.8 0:03.59 ruby
3	3460 td-agent 20 0 421m 168m 2664 S 0.0 9.0 0:06.57 ruby
4	3460 td-agent 20 0 385m 143m 2664 S 0.0 7.7 0:08.46 ruby
5	3460 td-agent 20 0 417m 205m 2664 S 0.0 10.9 0:10.47 ruby
6	3460 td-agent 20 0 549m 221m 2664 S 0.0 11.8 0:11.92 ruby
7	3460 td-agent 20 0 537m 220m 2664 S 0.0 11.7 0:13.53 ruby
8	3460 td-agent 20 0 537m 239m 2664 S 0.0 12.8 0:15.72 ruby
9	3460 td-agent 20 0 541m 264m 2664 S 0.0 14.1 0:17.55 ruby
10	3460 td-agent 20 0 545m 300m 2664 S 0.3 16.0 0:19.96 ruby

msgpack (0.5.8)にしてみた (secure-forward)

タイミング	結果(top)
0	3766 td-agent 20 0 229m 28m 3208 S 0.0 1.5 0:00.26 ruby
1	3766 td-agent 20 0 279m 59m 3592 S 0.0 3.2 0:13.44 ruby
2	3766 td-agent 20 0 295m 75m 3592 S 0.0 4.0 0:21.46 ruby
3
4
5
6
7
8
9
10