elk.md

A fast guide for installing an ELK system. Extracted from Install ELK in 14.04

Install Java 8

sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get update
sudo apt-get -y install oracle-java8-installer

Install Elasticseach

wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
echo 'deb http://packages.elasticsearch.org/elasticsearch/1.4/debian stable main' | sudo tee /etc/apt/sources.list.d/elasticsearch.list
sudo apt-get update
sudo apt-get -y install elasticsearch=1.4.4

# Congigure Elasticsarch
sudo vim /etc/elasticsearch/elasticsearch.yml

# To restrict outside access add this find `network.host` 
# and leave it this way `network.host: localhost`

# Start the Elasticsearch instance:
sudo service elasticsearch restart

# To start Elasticsearch on boot up:
sudo update-rc.d elasticsearch defaults 95 10

Install Kibana 4

cd ~; wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz
tar xvf kibana-*.tar.gz
vim ~/kibana-4*/config/kibana.yml
# In the configuration file, replace host: "0.0.0.0" for host: "localhost"

# Move kibana to /opt
sudo mkdir -p /opt/kibana
sudo cp -R ~/kibana-4*/* /opt/kibana/

# Install Kibana init script to run it as a service
cd /etc/init.d && sudo wget https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/bce61d85643c2dcdfbc2728c55a41dab444dca20/kibana4
sudo chmod +x /etc/init.d/kibana4
sudo update-rc.d kibana4 defaults 96 9
sudo service kibana4 start

Install NGINX

sudo apt-get install nginx apache2-utils

# Create an httpasswd file for protecting the Kibana instance:
sudo htpasswd -c /etc/nginx/htpasswd.users kibanaadmin

sudo vim /etc/nginx/sites-available/default
# Leave the nginx default server like the following example, changin the `server_name` value

server {
    listen 80;

    server_name example.com;

    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.users;

    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;        
    }
}

# Restar NGINX
sudo service nginx restart

# Kibana should be accessible from the domain specified in the `server_name` value

Install Logstash

echo 'deb http://packages.elasticsearch.org/logstash/1.5/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash.list
sudo apt-get update
sudo apt-get install logstash

Configure Logstash with SSL Certificates

sudo mkdir -p /etc/pki/tls/certs
sudo mkdir /etc/pki/tls/private

If you don't have a DNS use this for configure with the IP address.

sudo vim /etc/ssl/openssl.cnf
# Add this under [ v3_ca ] and replace the `logstash_server_private_ip` with your IP
subjectAltName = IP: logstash_server_private_ip

Generates the SSL certificates

cd /etc/pki/tls
sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

Configure Logstash

sudo vim /etc/logstash/conf.d/01-lumberjack-input.conf

Put this in the open file

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

Creates a syslog config file

sudo vim /etc/logstash/conf.d/10-syslog.conf

With the following filter inside

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

And the Lumberjack output

sudo vim /etc/logstash/conf.d/30-lumberjack-output.conf

With this inside:

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

# Start logstash server finally
sudo service logstash restart

Install logstash-forwarder in Client Servers

First copy your generated SSL certificate from your ELK to the client server

scp /etc/pki/tls/certs/logstash-forwarder.crt user@server_private_IP:/tmp

Then, in the client server:

sudo mkdir -p /etc/pki/tls/certs
sudo cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/

And:

echo 'deb http://packages.elasticsearch.org/logstashforwarder/debian stable main' | sudo tee /etc/apt/sources.list.d/logstashforwarder.list
wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get update
sudo apt-get install logstash-forwarder

Configure logstash-forwarder

sudo vim /etc/logstash-forwarder.conf

Find the network section and add this below

"servers": [ "logstash_server_private_IP:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"

Under files section add this between the sqare brackets

    {
      "paths": [
        "/var/log/syslog",
        "/var/log/auth.log"
       ],
      "fields": { "type": "syslog" }
    }

Restart the service

sudo service logstash-forwarder restart

Jpuelpan/elk.md