Anton Strukov Savemech

High availability monitoring with Sensu

sensu | May 8, 2013

Redundancy. Availability. Scalability. Should sound familiar to you if you work in the web. Every system I build has to fit those 3 main criterias. I would also throw in manageability. If I can’t use Chef with it, I’m probably trying to use the wrong tool for the job.

There’s always been one exception though: my monitoring tool. Nagios. Zabbix. Zenoss. Shinken. Used them all. Each of them have shortcomings when it comes to the four criterias listed above. Different ones for each.

So, that said, a few months back, I was searching for something fresh. Something I could easily manage with Chef. Because I tend to forget things, and I wanted to automate as much as possible our monitoring solution. Don’t get me wrong, there’s nothing as good as developers to monitor your stuff, but I tend to like to know there’s something wrong before they show up at my desk. Even if it’s only 1-2 minutes ;)

Sample service script for debianoids

Look at LSB init scripts for more information.

Usage

Copy to /etc/init.d:

# replace "$YOUR_SERVICE_NAME" with your service's name (whenever it's not enough obvious)

	#!/usr/bin/env python
	import json
	from jinja2 import Template
	# git clone https://github.com/pingcap/tidb-docker-compose
	# cd tidb-docker-compose
	# git clone https://github.com/tennix/grafonnet-lib -b table
	# python dashboard-to-jsonnet.py > pd.jsonnet
	# jsonnet -J grafonnet-lib pd.jsonnet > config/dashboards/generated-pd.json
	with open('config/dashboards/pd.json', 'r') as f:
	data = json.load(f)

	func main() {
	// Set logging output to standard console out
	log.SetOutput(os.Stdout)

	sigs := make(chan os.Signal, 1) // Create channel to receive OS signals
	stop := make(chan struct{}) // Create channel to receive stop signal

	signal.Notify(sigs, os.Interrupt, syscall.SIGTERM, syscall.SIGINT) // Register the sigs channel to receieve SIGTERM

	wg := &sync.WaitGroup{} // Goroutines can add themselves to this to be waited on so that they finish

	#!/bin/sh
	### BEGIN INIT INFO
	# Provides: consul
	# Required-Start: $local_fs $remote_fs
	# Required-Stop: $local_fs $remote_fs
	# Default-Start: 2 3 4 5
	# Default-Stop: S 0 1 6
	# Short-Description: Consul service discovery framework
	# Description: Healthchecks local services and registers
	# them in a central consul database.

	### Keybase proof

	I hereby claim:

	* I am Savemech on github.
	* I am savemech (https://keybase.io/savemech) on keybase.
	* I have a public key whose fingerprint is 33A8 D781 4CED C667 CEB6 656D 5B3F 2604 ED4E 867E

	To claim this, I am signing this object:

	---
	# This has been tested with ansible 1.3 with these commands:
	# ansible-playbook -i hosts ansible_conditionals_examples.yaml --extra-vars="hosts=myhosts isFirstRun=false"
	# ansible-playbook -i hosts ansible_conditionals_examples.yaml --extra-vars="hosts=myhosts isFirstRun=true"
	# ansible-playbook -i hosts ansible_conditionals_examples.yaml --extra-vars="hosts=myhosts"
	# NB: The type of the variable is crucial!

	- name: Ansible Conditionals Examples
	hosts: $hosts
	vars_files: