openslava2017_kafka.md

https://justpaste.it/openslava

filebeat (collection - lightweight) -> kafka (buffer) -> logstash (collection) -> elasticsearch (storage) -> kibana (display)

Kafka

Start Kafka:

bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
bin/kafka-server-start.sh -daemon config/server.properties

Produce and consume msgs from/to console:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

Produce and consume msgs from/to file

Edit file connect-file-source.properties:

name=local-file-source
connector.class=FileStreamSource
tasks.max=1
file=/tmp/myfile.txt
topic=test2

Edit file connect-file-sink.properties:

name=local-file-sink
connector.class=FileStreamSink
tasks.max=1
file=/tmp/myout.txt
topics=test2

bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties
echo Sunday >>/tmp/myfile.txt

Filebeat

filebeat.yml:

filebeat.prospectors:
- input_type: log
  paths:
    - /tmp/myfile.txt
  document_type: myexample
  exclude_files: ['\.gz$']
  ignore_older: 2m
  close_inactive: 1m
  fields:
     env: SCHOOL
  fields_under_root: true
  tail_files: true

output.kafka:
  hosts: ["localhost:9092"]
  topic: "test_log"
  partition.round_robin:
    reachable_only: false
  required_acks: 0
  compression: gzip
  max_message_bytes: 30000

filebeat-5.6.2-linux-x86_64 [] $ ./filebeat -c filebeat.yml