Trial Fluentd configuration for parsing HAProxy logs from Syslog

fluent.conf

<source>
  type tail
  path /var/log/haproxy.log
  pos_file /opt/fluentd/haproxy.log.post
  format /^(?<syslog_time>.+) (?<source_ip>.+) (?<ps>\w+)\[(?<pid>\d+)\]: (?<c_ip>[\w\.]+):(?<c_port>\d+) \[(?<time>.+)\] (?<f_end>[\w\.-]+) (?<b_end>[\w\.-]+)\/(?<b_server>[\w\.-]+) (?<tq>\d+)\/(?<tw>\d+)\/(?<tc>\d+)\/(?<tr>\d+)\/(?<tt>\d+) (?<status_code>\d+) (?<bytes_read>\d+) (?<captured_request_cookie>.+) (?<captured_response_cookie>.+) (?<termination_state>.+) (?<actconn>\d+)\/(?<feconn>\d+)\/(?<beconn>\d+)\/(?<srv_conn>\d+)\/(?<retries>\d+) (?<srv_queue>\d+)\/(?<backend_queue>\d+) "(?<http_request>.+)"$/
  tag debug.haproxy.http
  time_format %d/%B/%Y:%H:%M:%S
</source>

<match debug.**>
  type stdout
</match>

<match debug.**>
  type elasticsearch
  port 9200
  host localhost
  index_name fluentd
  type_name fluentd
  logstash_format true
  flush_interval 10s # for testing
</match>

Gemfile

source 'https://rubygems.org'

gem 'fluentd'
gem 'fluent-plugin-td'
gem 'fluent-plugin-elasticsearch'

Installation on SmartOS

# Installing fluentd on SmartOS

```
sudo rm /var/db/pkgin/pkgin.db
sudo pkgin update   # I hate you pkgin
sudo pkgin in ruby-2.1.2
ruby -v
```

Install gems
```
sudo gem install bundler
bundle --path vendor/bundle
```

Configure fluentd
```
cd /opt/fluentd
vi Gemfile
```

Setup and run!
```
bundle exec fluentd --setup .
bundle exec fluentd -c ./fluent.conf -vv
```

Particularly, HTTP default logs from HAProxy as defined in section 8.2.3. HTTP log format in http://www.haproxy.org/download/1.4/doc/configuration.txt