Skip to content

Instantly share code, notes, and snippets.

@ycombinator

ycombinator/mb_mon_migration.md

Last active December 27, 2019 16:06
Show Gist options
  • Save ycombinator/020a12c6e3c99d6c8d7b1bf783fef799 to your computer and use it in GitHub Desktop.
Save ycombinator/020a12c6e3c99d6c8d7b1bf783fef799 to your computer and use it in GitHub Desktop.
Download ZIP
Metricbeat monitoring of stack products: migration plans
Raw
mb_mon_migration.md

User perspective

Current state

As of: 7.5.0

  • ES:
    • Recommended: User installs and configures MB along side ES. MB monitors ES node and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Alternative: User configures ES to send monitoring data to itself (aka production cluster). Production cluster forwards monitoring data to monitoring cluster, .monitoring-* indices.
  • Kibana:
    • Recommended: User installs and configures MB along side Kibana. MB monitors Kibana instance and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Alternative: User configures Kibana to send monitoring data to production cluster. Production cluster forwards monitoring data to monitoring cluster, .monitoring-* indices.
  • Logstash:
    • Recommended: User installs and configures MB along side Logstash. MB monitors Logstash node and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Alternative: User configures Logstash to send monitoring data to production cluster. Production cluster forwards monitoring data to monitoring cluster, .monitoring-* indices.
  • Beats/APM server:
    • Recommended: User installs and configures MB along side Beat/APM server. MB monitors Beat/APM server instance and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Recommended: User configures Beat/APM server to send monitoring data directly to monitoring cluster, monitoring-* indices.
    • Deprecated (since 7.2.0): User configures Beat/APM server to send monitoring data to production cluster. Production cluster forwards monitoring data to monitoring cluster, .monitoring-* indices.

Products can ship directly to monitoring cluster, monitoring-* indices

Target: 7.7.0 (has to be before 8.0.0)

  • ES:
    • Recommended: User configures ES to send monitoring data directly to monitoring cluster, monitoring-* indices.
    • Alternative: User installs and configures MB along side ES. MB monitors ES node and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Deprecated: User configures ES to send monitoring data to itself (aka production cluster). Production cluster forwards monitoring data to monitoring cluster, .monitoring-* indices.
  • Kibana:
    • Recommended: User configures Kibana to send monitoring data directly to monitoring cluster, monitoring-* indices.
    • Alternative: User installs and configures MB along side Kibana. MB monitors Kibana instance and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Deprecated: User configures Kibana to send monitoring data to production cluster. Production cluster forwards monitoring data to monitoring cluster, .monitoring-* indices.
  • Logstash:
    • Recommended: User configures Logstash to send monitoring data directly to monitoring cluster, monitoring-* indices.
    • Alternative: User installs and configures MB along side Logstash. MB monitors Logstash node and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Deprecated: User configures Logstash to send monitoring data to production cluster. Production cluster forwards monitoring data to monitoring cluster, .monitoring-* indices.
  • Beats/APM server:
    • Recommended: User configures Beat/APM server to send monitoring data directly to monitoring cluster, monitoring-* indices.
    • Alternative: User installs and configures MB along side Beat/APM server. MB monitors Beat/APM server instance and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Deprecated (since 7.2.0): User configures Beat/APM server to send monitoring data to production cluster. Production cluster forwards monitoring data to monitoring cluster, .monitoring-* indices.

Metricbeat bundling - not sure we need this stage?

Target: 7.8.0

  • ES:
    • Recommended: Metricbeat packaged with ES. User configures Metricbeat. Metricbeat monitors ES node and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Alternative: User configures ES to send monitoring data directly to monitoring cluster, monitoring-* indices.
    • Deprecated: User installs and configures MB along side ES. MB monitors ES node and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Deprecated: User configures ES to send monitoring data to itself (aka production cluster). Production cluster forwards monitoring data to monitoring cluster, .monitoring-* indices.
  • Kibana:
    • Recommended: Metricbeat packaged with Kibana. User configures Metricbeat. Metricbeat monitors Kibana instance and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Alternative: User configures Kibana to send monitoring data directly to monitoring cluster, monitoring-* indices.
    • Deprecated: User installs and configures MB along side Kibana. MB monitors Kibana instance and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Deprecated: User configures Kibana to send monitoring data to production cluster. Production cluster forwards monitoring data to monitoring cluster, .monitoring-* indices.
  • Logstash:
    • Recommended: Metricbeat packaged with Logstash. User configures Metricbeat. Metricbeat monitors Logstash node and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Alternative: User configures Logstash to send monitoring data directly to monitoring cluster, monitoring-* indices.
    • Deprecated: User installs and configures MB along side Logstash. MB monitors Logstash node and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Deprecated: User configures Logstash to send monitoring data to production cluster. Production cluster forwards monitoring data to monitoring cluster, .monitoring-* indices.
  • Beats/APM server:
    • Recommended: User configures Beat/APM server to send monitoring data directly to monitoring cluster, monitoring-* indices.
    • Deprecated: User installs and configures MB along side Beat/APM server. MB monitors Beat/APM server instance and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Deprecated (since 7.2.0): User configures Beat/APM server to send monitoring data to production cluster. Production cluster forwards monitoring data to monitoring cluster, .monitoring-* indices.

Desired end state

Target: when Agent becomes GA, sometime in 8.x

  • ES:
    • Recommended: Agent packaged with ES. User configures Agent. Agent monitors ES node and sends monitoring data directly to monitoring cluster, monitoring-* indices.
  • Kibana:
    • Recommended: Agent packaged with ES. User configures Agent. Agent monitors Kibana instance and sends monitoring data directly to monitoring cluster, monitoring-* indices.
  • Logstash:
    • Recommended: Agent packaged with ES. User configures Agent. Agent monitors LS node and sends monitoring data directly to monitoring cluster, monitoring-* indices.
  • Beats:
    • Recommended: User installs and configures Agent. Agent monitors underlying Beats/APM server instance(s) and sends monitoring data directly to monitoring cluster, monitoring-* indices.

Development perspective

Foundation

Target: 7.6.0

  • Kibana Stack Monitoring UI and Telemetry code can read from .monitoring-* as well as monitoring-* indices.
  • ES creates resources (index templates, cluster alert watches, ingest pipelines) for .monitoring-* as well as monitoring-* indices. Resources are created eagerly.

Monitored product changes

Target: 7.7.0

  • ES exposes settings to send monitoring data directly to monitoring cluster, monitoring-* indices.
  • Kibana exposes settings to send monitoring data directly to monitoring cluster, monitoring-* indices.
  • Logstash exposes settings to send monitoring data directly to monitoring cluster, monitoring-* indices.

Monitored product changes - not sure we need this stage?

Target: 7.8.0

  • ES bundles Metricbeat with it.
  • Kibana bundles Metricbeat with it.
  • Logstash bundles Metricbeat with it.

Agent GA

Target:

  • ES bundles Agent with it.
  • Kibana bundles Agent with it.
  • Logstash bundles Agent with it.
  • Beats/APM server are controlled by Agent.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment