Java Application Monitoring.md

1 Java Application Monitoring Architecture

2 Components:

2.1 Application Metric monitoring agent StageMonitor:

An open source solution to application performance monitoring for java server applications. Its application is divided in two main parts:

Core
Plugins

Core is the main application which joins all parts of collection and sending data to storage database for monitoring and analysis. By default, core having web browser widget in it which is default enabled and will be seen if you are running application in serverlet api 3.0 and above.

2.1.1 Add StageMonitor to your existing Spring Application made in Maven Project

It’s a simple task of adding StageMonitor to your existing Spring Application which is a maven project by simply adding pom dependency.

<project>
  ...
  <dependencies>
    ...
    <dependency>
      <groupId>org.stagemonitor</groupId>
      <artifactId>stagemonitor-web</artifactId>
      <version>[VERSION]</version>
    </dependency>
    ...
  <dependencies>
  ...
</project>

More details: https://github.com/stagemonitor/stagemonitor/wiki/Step-1%3A-In-Browser-Widget

2.1.2 StageMonitor Settings: Setting file for stagemonitor is placed in resource folder, if it is not created then please create

stagemonitor.elasticsearch.url=http://192.168.94.51:9200
stagemonitor.web.widget.enabled=false
stagemonitor.alerts.frequency=10
stagemonitor.requestmonitor.http.requestparams.confidential.regex=(?i).*pass.*, (?i).*credit.*, (?i).*pwd.*
stagemonitor.instrument.include=org.springframework.samples.petclinic
stagemonitor.reporting.interval.elasticsearch=60
stagemonitor.instrument.exclude=org.springframework.samples.petclinic.model
#stagemonitor.password=
tagemonitor.web.rum.enabled=false
stagemonitor.alerts.checks={"c44d96fc-cbba-4d35-c4d0-6133d8c3c6e2"\:{"id"\:"c44d96fc-cbba-4d35-c4d0-6133d8c3c6e2","application"\:"Spring PetClinic","name"\:"Response Time","alertAfterXFailures"\:"1","metricCategory"\:"TIMER","target"\:"response_time_server.Process-Find-Form.total","thresholds"\:{"WARN"\:[{"metric"\:"p50","operator"\:"GREATER","thresholdValue"\:"1000"},{"metric"\:"p95","operator"\:"GREATER","thresholdValue"\:"5000"}],"ERROR"\:[],"CRITICAL"\:[]},"active"\:true}}
stagemonitor.requestmonitor.external.onlyReportNExternalRequestsPerMinute=1000000

Please remember stagemonitor.web.widget.enabled is used to disable web widget and is only working when we comment stagemonitor.password

Do remember to add a file named stagemonitor.properties in $App_Folder/WEB-INF/classes/stagemonitor.properties

2.2 ElasticSearch: Data Collector

Elasticsearch is Open Source, Distributed, RESTful Search Engine built for the cloud. It mainly used for storage of data and retrieval on demand. We used elasticsearch to store our monitoring data so that we can make analysis on it.

2.2.1 Install Elastic Search

Download appropriate file from https://www.elastic.co/downloads/elasticsearch
I am expecting to be debian(or Ubuntu), then download DEB format file. You can use wget to install on server like

wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.4.0/elasticsearch-2.4.0.deb

Install downloaded file from dpkg -i <downloaded_file>
Start the elastic search server by service service elasticsearch start
Now you can see service is started on default elasticsearch port 9200.

2.2.2 Map stageMonitor to elastic search

Map stagemonitor to elastic search is done by editing stagemonitor property file placed in $App_Folder/WEB-INF/classes/stagemonitor.properties . There are two variables which have to be change to specify elastic search

stagemonitor.elasticsearch.url=http://localhost:9200
stagemonitor.reporting.interval.elasticsearch=60

2.3 Kibana: Visualizer

Kibana is a visualizing tool in which you can create your custom graphs, visualizers, dashboard, etc. for monitoring/analysis of data store in many time based database like elasticsearch, etc.

2.3.1 Install Kibana

Download latest appropriate package of your OS from https://www.elastic.co/downloads/kibana and install on the system. More details found on https://github.com/stagemonitor/stagemonitor/wiki/Step-2%3A-Request-Analysis-Dashboard

3 Setting Elasticsearch and Kibana by Docker

You can also start elasticsearch and kibana through the help of docker-compose.

Install docker and docker-compose with the help of following links https://docs.docker.com/engine/installation/ and https://docs.docker.com/compose/install/
Now create new folder and make a file docker-compose.yml:

elasticsearch:
  image: elasticsearch:2.3.3
  ports:
   - "9200:9200"
   - "9300:9300"
  volumes:
  # adjust the path before the colon to some local directory where the data should be stored at
   - /var/data/docker/elasticsearch:/usr/share/elasticsearch/data
  # environment:
  # - ES_HEAP_SIZE=6g # adjust heap size, should be half of your nodes RAM (max 30g)
  command: elasticsearch
    --cluster.name monitoring-cluster
    --index.number_of_replicas 0
    --network.bind_host _non_loopback:ipv4_
    --node.name es-monitoring-01
    --http.cors.enabled true
    --http.cors.allow-origin *
    ##  only needed if you want to cluster elasticsearch across multiple hosts
    #--network.publish_host <ip of docker host>
    #--discovery.zen.ping.unicast.hosts <ip of other ES hosts>

    ##  Note:
    #   set 'node.box_type hot' for your beefy nodes with SSDs which hold the new indices and 'node.box_type cold' for historical nodes (see https://github.com/stagemonitor/stagemonitor/wiki/Elasticsearch#hot-cold-architecture)
    #   increase 'index.number_of_replicas' if you want backups (needs more disk space)
    #   don't forget to change the 'node.name' if you want multiple instances
  log_driver: "json-file"
  log_opt: # log rotation
    max-size: "10m"
    max-file: "20"
kibana:
  image: kibana:4.5.1
  ports:
   - "5601:5601"
  links:
   - elasticsearch:elasticsearch
  log_driver: "json-file"
  log_opt: # log rotation
    max-size: "10m"
    max-file: "20"

Now run docker-compose up to start container. If you want to start in background then run docker-compose up -d

4 Elastalert: Get Alert on Specific event

ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch. For further study please look https://github.com/Yelp/elastalert

4.1 Setting Up custom alert on error

Get pull from https://github.com/Yelp/elastalert.git
Please rename config.yaml.example to config.yaml and make changes according to infra need.

# Rule folder in which all rules need to specified for alerting
rules_folder: /home/vagrant/tmp/elastalert/rules

# ElasticSearch Host
es_host: 192.168.94.51

# Elastic Search Port 
es_port: 9200

writeback_index: elastalert_status

# For kibana
use_kibana4_dashboard: http://192.168.94.51:5601/app/kibana#/dashboard/elastalert

In above replace 192.168.94.51 with Ip address of elastic search.

Now create a rule in above specified rule folder e.g., rules/java_error.yaml

# Name of alert
name: Java Error application

# type of alert, currently we need to 
type: frequency

# Search index in Elasearch
index: stagemonitor-requests-*
num_events: 1
timeframe:
  minutes: 5
alert_on_new_data: True
filter:
  - term:
      error: "true"

# Get alert on following. 
alert:
  - email
  - slack

# email smtp configuration to  send emails                                                                                                                                                                                                                                                                                                                                                              
smtp_host: "email-smtp.us-west-2.amazonaws.com"
smtp_port: "587"
smtp_auth_file: "/home/vagrant/tmp/elastalert/smtp.auth.yml"
email_reply_to: "[email protected]"
from_addr: "[email protected]"
alert_subject: "Issue in {0} occurred at {1}"
alert_subject_args:
- application
- "@timestamp"

# Slack webhook config
slack_webhook_url: "https://hooks.slack.com/services/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"
slack_username_override: "Test_Watcher"
slack_channel_override: "#timepass"


email:
  - [email protected]

The above alert is configured to get 1 event in 5 minutes of index stagemonitor-requests* when error data will be true

5 References:

StageMonitor https://github.com/stagemonitor/stagemonitor/wiki
Track your own metrics https://github.com/stagemonitor/stagemonitor/wiki/Track-your-own-metrics
ElasticSearch https://www.elastic.co/products/elasticsearch
ElasticSearch More https://www.elastic.co/products
Kibana https://www.elastic.co/products/kibana
Elastalert https://elastalert.readthedocs.io/en/latest/
Docker https://docs.docker.com/
Docker Compose https://docs.docker.com/compose/

varunpalekar/Java Application Monitoring.md