Monitoring, Log Management and Visualization Tools Cheatsheet

Logstash.md

LogStash

Logstash

• Originally started off as a log collector. Has become unifying point for disparate data sources.
• LS serves as the data ingestion point for ElasticSearch (ES).

Dead Letter Queues

• "Dead Letter Queues" in Logstash protect against data loss - ref
• When Logstash encounters an unprocessable event, it drops the unsuccessful event by default.
• To prevent data loss you can send it to a DLQ (only supported when outputting to ES).
• Add an additional pipeline to process this bad data and send it to ElasticSearch.

Prometheus.md

Prometheus

• Primary purpose is metrics collection from applications and services.
• Prometheus allows gathering of multi-dimensional data ( what is dimensional data? )

I like to pronounce it with a British accent - Pro-mee-fi-uss.

Key	Value
Types of Metrics
Counter	for any value that does not decrease.
Gauge	for values that can increase or decrease.
Histogram
Summary

❓ How is Dimensional Data enabled in Prometheus?

ref

• All metrics are gathered as time-stamped values ("time series")
• Each metric has accompanying labels.

Instrumentation

• Service monitoring is achieved by instrumenting services with appropriate Prometheus client library. This library allows you to
• configure metrics to be collected
• expose collected metrics on an endpoint (e.g /prometheus)
• A Prometheus Server pulls metrics periodically from above exposed endpoints.

Service Discovery

• You can hard-code the server hostnames into Prometheus config or use "service discovery" which dynamically discovers targets to scrape metrics from.

Prometheus Storage

• Prometheus stores Time Series Data into "shards" on local storage.
  • 💡 A shard is an individual block or partition of a large database. Shards are usually stored in a distributed fashion to spread the load.
• Each shard or block is of 2 hours worth of data.

❗ Note that a limitation of the local storage is that it is not clustered or replicated. Thus, it is not arbitrarily scalable or durable in the face of disk or node outages and should thus be treated as more of an ephemeral sliding window of recent data. Prometheus local storage is not meant to be durable ( ref )

• Prometheus + Remote Storage (e.g ElasticSearch) for more durability and scalability.

❓ Can Prometheus be compared to Logstash?

ref

• Prometheus has an inbuilt Time Series DB but Logstash does not (however, ElasticSearch can serve as the TS DB for LS).
• Prometheus monitors services and gathers metrics.
• Logstash gathers a lot of data including metrics, from various sources including Log Files.
• Logstash gathers Metrics via the Beats framework.
• Metricbeats module for Prometheus allows Prometheus metrics to be gathered by Logstash.

Whereas Prometheus is primarily used to gather data about the health of an application or a system, Logstash can be used for gathering a wider variety of data - from sources as disparate as Twitter feeds, arbitrary HTTP requests etc. There is also evidence that Logstash can store/transform data from IoT devices 😲

Ref:

1. Prometheus Integration with ElasticSearch - Prometheusbeat, Metricbeat module for Prometheus