ElasticsearchとKibanaを使ってみる

elasticsearch.md

ElasticsearchとKibanaを使ってみる

Oracle JDKのインストール

$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer

Elasticsearchの実行

ダウンロードと起動

最新版を取ってくる（http://www.elasticsearch.org/overview/elkdownloads/）

$ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.tar.gz
$ tar xzfv elasticsearch-1.4.2.tar.gz

起動

$ ./elasticsearch-1.4.2/bin/elasticsearch

Pythonでデータ入力

elasticsearch-pyをインストール

$ sudo apt-get install python-pip
$ sudo pip install elasticsearch

標準入力からJSONを読み込んで、ひたすら入力するPythonスクリプト、esstore.py

import argparse
import json
from datetime import datetime
from elasticsearch import Elasticsearch

class EStore:
    def __init__(self, args):
        self.es = Elasticsearch(args.server)
        
        self.index   = args.index
        self.type    = args.type
        self.verbose = args.verbose

        print(self.verbose)

    def run(self):
        while True:
            line = input()
            data = json.loads(line)
            data['timestamp'] = datetime.now()
            self.es.index(index=self.index, doc_type=self.type, body=data)

            if self.verbose:
                print(data)

def parse_args():
    parser = argparse.ArgumentParser(description='store JSON to Elasticsearch')
    
    parser.add_argument('-s', dest='server', default=['localhost:9300'], nargs='+',
                        help='server address to Elasticsearch (default = localhost:9300)')
    parser.add_argument('-i', dest='index', required=True,
                        help='index for Elasticsearch')
    parser.add_argument('-t', dest='type', required=True,
                        help='type for Elasticsearch')
    parser.add_argument('-v', dest='verbose', default=False, action='store_true',
                        help='enable verbose mode')

    return parser.parse_args()

def main():
    args = parse_args()

    es = EStore(args)
    es.run()

if __name__ == '__main__':
    main()

実行。-i は index、-t は type を指定。index、type はそれぞれMySQLでいうデータベースとテーブルに相当。

$ python3 esstore.py -i test -t test -s localhost
{"foo": 1}
{"bar": 2}

入力結果をブラウザで見てみる。

http://localhost:9200/test/_search?pretty=true&q=*:*&size=1000

{
  "took" : 8,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "test",
      "_type" : "test",
      "_id" : "AUtyoEWeqMeExsCqQhUj",
      "_score" : 1.0,
      "_source":{"foo": 1, "timestamp": "2015-02-10T17:35:15.927973"}
    }, {
      "_index" : "test",
      "_type" : "test",
      "_id" : "AUtyoGwJqMeExsCqQhUk",
      "_score" : 1.0,
      "_source":{"bar": 2, "timestamp": "2015-02-10T17:35:25.884416"}
    } ]
  }
}

indexの内容を全部削除。

$ curl -XDELETE 'http://localhsot:9200/test'

mappingの設定

現在のmappingを確認

http://localhost:9200/test/_mapping/test?pretty=true

mappingを設定

$ curl -XPUT 'http://localhost:9200/test/_mapping/test?ignore_conflicts=true' -d '{
  "test" : {
    "properties" : {
      "client" : {
        "properties" : {
          "header" : {
            "properties" : {
              "host" : {
                "type" : "string",
                "index": "analyzed",
                "analyzer" : "simple",
                "fields" : {
                  "raw" : {"type" : "string", "index" : "not_analyzed"}
                }
              },
              "referer" : {
                "type" : "string",
                "index": "analyzed",
                "analyzer" : "simple",
                "fields" : {
                  "raw" : {"type" : "string", "index" : "not_analyzed"}
                }
              }
            }
          },
          "method" : {
            "properties" : {
              "uri" : {
                "type" : "string",
                "index" : "analyzed",
                "analyzer" : "simple",
                "fields" : {
                  "raw" : {"type" : "string", "index" : "not_analyzed"}
                }
              }
            }
          }
        }
      }
    }
  }
}'

Kibanaのインストールと設定

$ cd /var/www/html
$ sudo wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.2.tar.gz
$ sudo tar xzfv kibana-3.1.2.tar.gz
$ sudo ln -s kibana kibana-3.1.2
$ sudo vi kibana/config.js

elasticsearch: "http://localhost:9200",

KibanaからElasticsearchにアクセス出来るように設定を変更

$ cd ~/elasticsearch-1.4.2/
$ vi config/elasticsearch.yml

http.cors.enabled: true

ブラウザからアクセス

http://localhost/kibana/