elasticsearch with cluster setup

Simple Notes understand

logstash with elasticsearch with redis
=======================================

Since the last to make the configuration hoards logstash by directly loading the Apache log on the same server, as a next step, this time I try to make logstash the Apache configuration to collect logs of multiple servers.
And ... I can also say, try it first on a single server.

Official page of Centralized Setup with Event Parsing referring to.

And data flow components that make it work, like this.
Figure.1 logstash Architecture
Installing Redis
First install the Redis.

Download the source Redis to tmp /redis.
-----------------------------------------

[root@server]# cd /tmp
[root@server]# mkdir redis
[root@server]# cd redis
[root@server]# wget http://redis.googlecode.com/files/redis-2.4.17.tar.gz
[root@server]# tar xvzf redis-2.4.17.tar.gz
redis-2.4.17/
redis-2.4.17/.gitignore
(snip)

to make.

[root@server]# cd redis-2.4.17
[root@server]# make
cd src && make all
make[1]: 
`/tmp/redis/redis-2.4.17/src'
(snip)


Start the Redis.

root@server]# cd /opt/redis
[root@server]# src/redis-server
[27291] 12 Sep 12:58:43 - 0 clients connected (0 slaves), 567120 bytes in use


launched (the role you want to transfer the logs to the parent server) shipper
Next, implement the ability to send to collect the log of Redis Apache.
This log is intended for Apache on the same server, you can also take place in a different server if the work carried out in this section, are also collected Apache log on the server that should be possible.

First, create a configuration file logstash.
--------------------------------------------

[root@server]# cd /opt/logstash
[root@server]# vi shipper.conf
shipper.conf
------------
	
input {
  file {
    type => "apache"
    path => "/var/log/httpd/access_log"
  }
}
 
filter {
  grok {
    type => "apache"
    pattern => "%{COMBINEDAPACHELOG}"
  }
}
 
output {
  stdout { debug => true debug_format => "json"}
  redis { host => "127.0.0.1" data_type => "list" key => "logstash" }
}


Try to start.

Then, the standard output of Redis

Labeled. Since the number of clients increased to 1 0, shipper and I was able to connect to the Redis seems alright.
launched indexer (role to be written to the DB receives the data from Redis)
Centralized Setup with Event Parsing example has been written and installed separately (ElasticSearch) DB. But this is a simple test, I decided to use ElasticSearch that is bundled with the logstash.

[root@server]# cd /opt/logstash
[root@server]# vi indexer.conf

indexer.conf
------------

input {
  redis {
    host => "127.0.0.1"
    type => "redis-input"
    data_type => "list"
    key => "logstash"
    message_format => "json_event"
  }
}
 
output {
  stdout { debug => true debug_format => "json"}
 
  elasticsearch {
     embedded => true
#    host => "127.0.0.1"
  }
}

Write a startup script.

[root@server]# vi indexer.sh

indexer.sh
----------

java -jar logstash-1.1.1-monolithic.jar agent -f indexer.conf

Try to start.

[root@server]# chmod +x indexer.sh
[root@server]# ./indexer.sh

Then, the standard output of Redis
[27291] 12 Sep 13:00:29 - 2 clients connected (0 slaves), 567120 bytes in use
	
Labeled. Since the number of clients increased to two, I was able to connect to the Redis safely as well as indexer.

In this state, try to access appropriately to Apache in your browser.
Then, the standard output of the indexer

{"@source":"file://server/var/log/httpd/access_log","@type":"apa
che","@tags":[],"@fields":{"clientip":["xx.xx.xx.xxx"],"ident":[
"-"],"auth":["-"],"timestamp":["12/Sep/2012:13:01:32 +0900"],"ZO
NE":["+0900"],"verb":["GET"],"request":["/images/true.png"],"htt
pversion":["1.1"],"response":["200"],"bytes":["248"],"referrer":
["http://server/themes/farend_basic/stylesheets/application.css?
1279358964"],"agent":["\"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:
15.0) Gecko/20100101 Firefox/15.0.1\""]},"@timestamp":"2012-09-1
2T04:01:32.889000Z","@source_host":"server","@source_path":"/var
/log/httpd/access_log","@message":"xx.xx.xx.xxx - - [12/Sep/2012
:13:01:32 +0900] \"GET /images/true.png HTTP/1.1\" 200 248 \"htt
p://server/themes/farend_basic/stylesheets/application.css?12793
58964\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/201
00101 Firefox/15.0.1\""}

Labeled. It is likely to apparently working well. Launch the Web GUI
Finally, start the Web GUI for the search log.
Write a startup script.

[root@server]# cd /opt/logstash
[root@server]# vi web.sh
	
web.sh
-------
java -jar logstash-1.1.1-monolithic.jar web --backend elasticsearch:///?local

Try to start.
So, to access the http://server:9292/, 
-------------------------------------

I'll try to find suitably.  logstash WebUI
When the search results can be displayed!


I collected the log of Apache in logstash1.1.1

Following the recent fluentd + MongoDB, I've tried logging solutions & search OSS, the logstash.  Assumptions and SpecificationsThis time the goal is for the time being, the search logs to collect Apache on the same server.
Environmental CentOS 5.5.
Installing logstash

logstash - open source log management download logstash-1.1.1-monolithic.jar from and install it. That said, I just put the jar in a place of your choice. Directory tried to /opt/logstash.

[root@server]# cd /opt/logstash/
[root@server]# ls
logstash-1.1.1-monolithic.jar

Logstash tutorial @ Dreamforce of
Step 6 - real world example + search
With reference to the apache-elasticsearch.conf located, create a configuration file mylogstash.conf.
Examples of the above site, which had become a form of a log of apache Kuwaseru in nc command event, we should try a stand-alone for the time being, I was directly specify the log file. I have also commented out in my environment, so parsing the date did not go well (in error).

[root@server]# vi mylogstash.conf

mylogstash.conf
---------------

input {
  file {
    type => "apache"
    path => "/var/log/httpd/access_log"
  }
  #tcp {
  #  type => "apache"
  #  port => 3333
  #}
}
 
filter {
  grok {
    type => "apache"
    pattern => "%{COMBINEDAPACHELOG}"
  }
 
# 2012.09.11 dsp74118 comment-out due to parse error
#  date {
#    type => "apache"
#    timestamp => "dd/MMM/yyyy:HH:mm:ss Z"
#  }
}
 
output {
  stdout { debug => true debug_format => "json"}
  elasticsearch {
    embedded => true
  }
}

Create a script to launch the logstash.

[root@server]# vi start.sh

start.sh
---------

java -jar logstash-1.1.1-monolithic.jar agent -f mylogstash.conf -- web --backend elasticsearch:///?local

Try to start.

[root@server]# ./start.sh
parse
logfile
thread
remaining
log4j:WARN No appenders could be found for logger (org.elasticsearch.node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

log4j error came out. Because the cause is log4j configuration file that did not make it into jar file, create a log4j.properties as follows.
log4j.properties

[root@server]# jar -uf logstash-1.1.1-monolithic.jar log4j.properties
---------------------------------------------------------------------
	
Use the jar command to store the log4j.properties file in the jar.

[root@server]# ./start.sh
parse
logfile
thread
remaining

Log4j errors are eliminated.
Or try log can be collected
Try to access the Apache PC browsers of the neighborhood.

…
{"@source":"file://server/var/log/httpd/access_log","@type":"apa
che","@tags":[],"@fields":{"clientip":["xx.xx.xx.xx"],"ident":["
-"],"auth":["-"],"timestamp":["11/Sep/2012:13:14:56 +0900"],"ZON
E":["+0900"],"verb":["GET"],"request":["/javascripts/effects.js?
1290948531"],"httpversion":["1.1"],"response":["304"],"referrer"
:["http://server/issues/252"],"agent":["\"Mozilla/5.0 (Windows N
T 6.1; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0.1\""]},"@time
stamp":"2012-09-11T04:25:27.121000Z","@source_host":"server","@s
ource_path":"/var/log/httpd/access_log","@message":"xx.xx.xx.xxx
 - - [11/Sep/2012:13:14:56 +0900] \"GET /javascripts/effects.js?
1290948531 HTTP/1.1\" 304 - \"http://server/issues/252\" \"Mozil
la/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20100101 Firefox/1
5.0.1\""}
…


log logstash has collected is displayed on the standard output.
I'll try to find the log in the Web GUI
I then take a look at the GUI.


http://server:9292/ access to a browser.
------------------------------------------
Figure.1 logstash search screen
Because the search screen appears, click [Search] to enter a search keyword appropriately.
Search Results screen Figure.2
When the search results appear.
I'll try to specify a range of time. Feel like following.
@ Timestamp <Search>: [YYYY-MM-DDTHH: MI: SS TO YYYY-MM-DDTHH: MI: SS]

Where I've narrowed down the time Figure.3
Expected results were obtained.

Next time I want to collect any other Apache server logs.