install the ELK stack on an ubuntu-x64 machine on Azure

This guide has been revised 06-03-2015. Start with a clean ubuntu 14.04LTS-x64 machine and get it updated

sudo su
apt-get update && apt-get dist-upgrade
reboot

create additional data disk in Azure

start with on: http://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-how-to-attach-disk/ mount the disc as '/elasticsearch-data

mkdir /elasticsearch-data/data
mkdir /elasticsearch-data/logs

#set the owner of the dirs to the elasticsearch service
chown elasticsearch.elasticsearch /elasticsearch-data/data
chown elasticsearch.elasticsearch /elasticsearch-data/logs

#keep away nosey people
chmod 700 /elasticsearch-data/data
chmod 700 /elasticsearch-data/logs

install elasticsearch

apt-get install openjdk-7-jre-headless -y

Now we can configure apt-get to listen to package update from elasticsearch. Add it to your apt-get command. Following steps are extracted from http://www.elasticsearch.org/blog/apt-and-yum-repositories/ NOTE: 1.2 was latest stable releast at this time (15-07-2014). Update the package source url as required

#get the GPG key for elasticsearch apt-get channel
wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add - 

#add package source to your sources.list
nano /etc/apt/sources.list
#nano opens the text editor
#add following lines to sources.list:
deb http://packages.elasticsearch.org/elasticsearch/1.4/debian stable main
#save file / exit nano

apt-get update && apt-get install elasticsearch

Elasticsearch is now up and running. Important paths to remember:

Configs are stored in /etc/elasticsearch/
Elasticsearch resides in /usr/share/elasticsearch/

More information: http://code972.com/blog/2014/07/74-the-definitive-guide-for-elasticsearch-on-windows-azure

start / stop the elasticsearch service:

/etc/init.d/elasticsearch restart
restart|stop|start

tweak config of elasticsearch

vi /etc/elasticsearch/elasticsearch.yml

cluster.name: my-production-cluster
node.name: "my-node-name"
#determine if you need script acces, it is a possible security risk!
script.disable_dynamic: false
#if you run 2 nodes set it to 2. This will make ES auto start the restore/balancing if 2 nodes are found
gateway.expected_nodes: 2
#if you use unicast, set the ip's of the other nodes in this array
discovery.zen.ping.unicast.hosts: ["10.0.0.1"]
#set your data + logs on an other partition
path.data: /elasticsearch-data/data
path.logs: /elasticsearch-data/logs

configure memory allowance of elasticsearch

vi /etc/init.d/elasticsearch

Add / update the following vars:

ES_MIN_MEM=256m
ES_MAX_MEM=1g

configure the max file descriptors

ElasticSearch uses quite a few of file descriptors, both for Lucene indexes and Netty. Raise the number of available file descriptors to the user running ElasticSearch. [http://www.elasticsearch.org/tutorials/too-many-open-files/]

vi /etc/security/limits.conf
# append the following to the file:
elasticsearch soft nofile 32000
elasticsearch hard nofile 32000

make elasticsearch service start on reboot of machine

update-rc.d elasticsearch defaults 95 10

install ES plugins

cd /usr/share/elasticsearch
bin/plugin --install mobz/elasticsearch-head

configure azure port forwarding

Last but not least; configure your VM to pass-through connections for your external dns name port 9200 (or whatever you like) to the internal VM 9200 port). Login in your Azure Management portal and navigate to your VM. Go to Endpoints and add an endpoint for port 9200 internal + external => ok => ok => ok. wait done!

install logstash (1.4.2)

partially taken from partial from http://blog.dimaj.net/content/howto-view-and-analyze-your-logs-web-page

cd ~ 
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
tar xzvf logstash-1.4.2.tar.gz 
mv logstash-1.4.2 /opt/logstash
adduser --system --disabled-login --no-create-home --group logstash 
usermod -a -G adm logstash

#auto start script for logstash
# download init script 
wget -O /etc/init.d/logstash https://raw.githubusercontent.com/elasticsearch/logstash/master/pkg/logstash.sysv
# make it executable 
chmod +x /etc/init.d/logstash 
# make it auto-startable 
update-rc.d logstash defaults

# configure logstash folder

# create logstash config dir 
mkdir -p /etc/logstash/conf.d 
# create logstash logs dir 
mkdir /var/log/logstash 
chown -R logstash: /var/log/logstash 
# make home folder 
mkdir /var/lib/logstash 
chown -R logstash: /var/lib/logstash

Place your logstash config file (myconfig.conf) in /etc/logstash/conf.d, it will be automatically picked up after you start the logstash service (service logstash start)

install kibana 4.0.1

Kibana 4 is no longer just fancy website but comes with a complete pre-packaged webserver.

    cd ~
    wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz
    tar xzvf kibana-4.0.1-linux-x64.tar.gz
    rm kibana-4.0.1-linux-x64.tar.gz
    mv kibana-4.0.1-linux-x64 /var/opt/kibana-4.0.1

configure kibana webserver port + elasticserach connection

    vi /var/opt/kibana-4.0.1/config/kibana.yml

auto-start kibana 4

There is no init.d script yet for kibana 4. :/

#start screen which keeps running even if you disconnect your session
screen
./bin/kibana

denmerc/installation.md