Route Docker Logs to ELK Stack

Instructions.md

• With Docker 1.8.0 shipped new log-driver for GELF via UDP, this means that the logs from Docker Container(s) can be shipped directly to the ELK stack for further analysis.
• This tutorial will illustrate how to use the GELF log-driver with Docker engine.
• Step 1: Setup ELK Stack:
  • docker run -d --name es elasticsearch
  • docker run -d --name logstash --link es:elasticsearch logstash -v /tmp/logstash.conf:/config-dir/logstash.conf logstash logstash -f /config-dir/logstash.conf
  • Note the config for Logstash can be found at this link
  • docker run --link es:elasticsearch -d kibana
• Once the ELK stack is up now let's fire up our nginx container which ships its logs to ELK stack.

• LOGSTASH_ADDRESS=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' logstash)
• docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://$LOGSTASH_ADDRESS:12201 --log-opt gelf-tag="fe" nginx
• All logs from the nginx container will be shipped to our ELK stack for slicing and dicing.
• To verify that logs are being passed in visit http://<kibana-container-ip>:5601 follow through the setup and you should see the logs in Kibana.

This works for me:
docker run -d --name es elasticsearch

docker run -d --name logstash --link es:elasticsearch logstash -v "$PWD":/config-dir -f /config-dir/logstash.conf

docker run --link es:elasticsearch -d kibana

docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://127.0.0.1:12201 --log-opt tag="fe" nginx

logstash.conf:

input {
gelf { }
}

output {
elasticsearch {
hosts => ["elasticsearch"]
}
stdout { }
}

Can someone explain, how not to hard-code elastic container ip in logstash.conf?

logstash container is restarting.

Logstatsh process inside the container

root@4e14335c9d93:/# ps -ef | grep logstash
logstash     1     0 47 13:25 ?        00:00:08 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb -v /home/vagrant:/config-dir -f /config-dir/logstash-gelf.conf

I queried the docker logs. I see logstash is not recognizing -v option

vagrant@PerfQual-host:~$ docker logs 4e14335c9d937874715d63c9a6413a33749913f1339f964c5dac8ef0cdb78426
ERROR: Unrecognised option '-v'

See: 'bin/logstash --help'
ERROR: Unrecognised option '-v'

See: 'bin/logstash --help'
vagrant@PerfQual-host:~$

I have written the docker-elk-deployment project to simplify these steps, it supports the Elastic Stack 5.2.0+ on swarm mode cluster, and use gelf logging driver to gathering logs from docker containers, anyone who has problems with these steps can go to https://github.com/HackerWilson/docker-elk-deployment and have a try.

Is there any way to increase the size of logs logstash can take? Logstash is breaking apart my logs into multiple messages which then fails to parse.

@AlecBruns see logstash-plugins/logstash-input-gelf#37 and moby/moby#22920 and moby/moby#22979

You can't use docker logging if you want to parse multiple lines.

I found that this does not work when using docker-compose has anyone else had luck? https://forums.docker.com/t/docker-loading-in-stack/33051

Can I use this to send logs directly in ELK and keep them in the journal of the host?

Here's a working verision which only requires docker-compose.
https://github.com/amalic/nginxelk

Please let me know what you think.