mlafeldt

Shakespeare Sonnet++ Postmortem (incident #465)

Date

2015-10-21

Authors

• jennifer
• martym

mlafeldt

• Read and understand https://help.github.com/articles/signing-commits-using-gpg/

• Configure Git to sign all the commits, see https://github.com/mlafeldt/dotfiles/commit/f4554f0a122145509da5cdf4180037b3bc13b0ab

• Install gpg-agent (we don't want to enter a password for each commit)

brew install gpg-agent

mlafeldt

wget https://apt.dockerproject.org/repo/pool/main/d/docker-engine/docker-engine_1.11.1-0~trusty_amd64.deb
sudo dpkg -i docker-engine_1.11.1-0~trusty_amd64.deb
sudo apt-get -y -f install

mlafeldt

Nomad Resilience Testing with Vagrant

Based on https://github.com/hashicorp/nomad/tree/master/demo/vagrant, I did some resilience testing to figure out how reliable Nomad's periodic jobs are.

Test scenario

I used this minimal periodic job, which executes a shell script every minute:

# example.nomad

mlafeldt

require "stringex"

desc "Create a new letter"
task :new_letter, :title do |t, args|
  title = args[:title] || "New Letter"
  num = File::basename(Dir['letters/*'].last)[0,3].to_i + 1
  filename = "letters/%03d-%s.md" % [num, title.to_url]
  date = Time.now.strftime("%Y-%m-%d")

  puts "==> Creating new letter: #{filename}"

mlafeldt

First of all, install Homebrew itself.

As the tap is a private Git repo, you need to generate a GitHub token with repo scope and then add this token to your ~/.netrc file like this:

machine github.com
  login <your GitHub user>
  password <your GitHub token>

mlafeldt

- name: "Build and push Docker images"
  sudo: yes
  command: "java -jar /opt/jenkins/jenkins-cli.jar -s http://127.0.0.1:8080 build {{ item }} -f"
  args:
    creates: "/var/lib/jenkins/jobs/{{ item }}/workspace"
  with_items:
    - CLI
    - Deployer
    - Foo
    - Bar

mlafeldt

Service metrics

I would strongly suggest having your services expose basic metrics themselves. At a bare minimum, for a web service you should probably expose metrics like response times and error rates—vital if your server isn’t fronted by a web server that is doing this for you. But you should really go further. For example, our accounts service may want to expose the number of times customers view their past orders, or your web shop might want to capture how much money has been made during the last day.

Semantic Monitoring

We can try to work out if a service is healthy by, for example, deciding what a good CPU level is, or what makes for an acceptable response time. If our monitoring system detects that the actual values fall outside this safe level, we can trigger an alert— something that a tool like Nagios is more than capable of.

However, in many ways, these values are one step removed from what we actually want to track—namely, is the system working? The more complex the interactions

mlafeldt

package main

import (
	"log"
	"os"

	"github.com/hashicorp/vault/api"
	"github.com/hashicorp/vault/builtin/credential/github"
)

mlafeldt

require 'stringex'

POSTS_DIR = '_posts'
BUILD_DIR = '_site'
DEPLOY_DIR = '_deploy'
DEPLOY_BRANCH = 'master'

def git(*args)
  sh 'git', *args
end