Creating a cookbook for our CI server

Requirements

Chef Development Kit (ver. 0.11.0)

Project setup

git clone https://github.com/joebew42/dropwizard-sample-app.git
cd dropwizard-sample-app/cookbooks/

Create the ci cookbook

chef generate cookbook ci

This will create a scaffold of the cookbook.

Our first recipe

recipes/default.rb

include_recipe 'java'

How can I try it ? test-kitchen (aka kitchen) is a tool used to run integration tests (serverspec) against a machine.

The kitchen workflow can described as follows:

create
setup
converge
verify
destroy

Let's initialize kitchen files and directories:

kitchen init

.kitchen.yml

---
driver:
  name: vagrant

provisioner:
  name: chef_zero
  client_rb:
    file_cache_path: '/var/chef/cache'

platforms:
  - name: ubuntu/trusty64
    driver:
      vagrantfile_erb: Vagrantfile

suites:
  - name: default
    run_list:
      - recipe[ci::default]
    attributes:

Vagrantfile

Vagrant.configure('2') do |config|
  config.vm.box = 'ubuntu/trusty64'
  config.vm.box_check_update = false
  config.vm.network :private_network, ip: '192.168.33.33'

  config.berkshelf.enabled = false

  if Vagrant.has_plugin?("vagrant-omnibus")
    config.omnibus.chef_version = 'latest'
  end

  if Vagrant.has_plugin?("vagrant-cachier")
    config.cache.scope = :box
    config.cache.auto_detect = false
    config.cache.enable :apt
    config.cache.enable :yum
    config.cache.enable :gem
    config.cache.enable :chef_gem
    config.cache.enable :generic, { :cache_dir => "/var/chef/cache" }
  end

  config.vm.provider "virtualbox" do |vb|
    vb.memory = 1024
  end
end

Now run kitchen create in order to create the machine used to run integration tests against

kitchen create

Apply chef cookbook:

kitchen converge

Ops! Some errors are thrown :/

Fix our recipe first

Adding java as cookbook dependecy by putting this line in metadata.rb:

depends 'java', '~> 1.39.0'

Put some java's specific attributes in attributes/java.rb

default['java']['jdk_version'] = '7'

Now try to run a converge:

kitchen converge

Java is installed! :)

How to verify that Java is installed ?

Here we use serverspec, that is a set of test helpers that can be executed against each kind of machine (manually or automatically provisioned). Useful for smoke tests.

Create our first integration test that verifies if java is correctly installed:

test/integration/default/serverspec/default_spec.rb

require 'serverspec'

set :backend, :exec

describe 'ci::default' do
  describe command('/usr/bin/java -version') do
    its(:stderr) { should contain('1.7') }
  end
end

Execute the verification with:

kitchen verify

All (just one) tests passes! :D

Put all things together

We want to install jenkins and configure it (ssh keys, directories, etc...)

Jenkins cookbook

Add the jenkins cookbook as dependency in metadata.rb:

depends 'java', '~> 1.39.0'
depends 'jenkins', '~> 2.4.1'

Update berkshelp dependencies with berks update.

Put the jenkins::master in default.rb recipe:

include_recipe 'java'
include_recipe 'jenkins::master'

run the converge with kitchen converge

Visits: http://192.168.33.33:8080 :D

Just one simple integration test

test/integration/default/serverspec/default_spec.rb

require 'serverspec'

set :backend, :exec

describe 'ci::default' do
  describe command('/usr/bin/java -version') do
    its(:stderr) { should contain('1.7') }
  end

  describe port(8080) do
    it { should be_listening }
  end
end

Run verification: kitchen verify

Complete the recipe

We are going to complete the recipe by installing maven and mysql

metadata.rb

depends 'java', '~> 1.39.0'
depends 'jenkins', '~> 2.4.1'
depends 'maven', '~> 2.1.1'
depends 'mysql', '~> 6.1.2'
depends 'mysql2_chef_gem', '~> 1.0.2'
depends 'database', '~> 4.0.9'

run berks update

recipes/default.rb

include_recipe 'java'
include_recipe 'maven'
include_recipe 'jenkins::master'

jenkins_plugin 'git'
jenkins_plugin 'greenballs'
jenkins_plugin 'junit'
jenkins_plugin 'jobConfigHistory'
jenkins_plugin 'delivery-pipeline-plugin' do
  notifies :restart, 'service[jenkins]', :delayed
end

package 'git'

mysql_service 'test' do
  port '3306'
  version '5.5'
  initial_root_password 'root'
  action [:create, :start]
end

mysql2_chef_gem 'default' do
  action [:install]
end

mysql_database 'db_notes_test' do
  connection(
    :host     => '127.0.0.1',
    :username => 'root',
    :password => 'root'
  )
  action :create
end

test/integration/default/serverspec/default_spec.rb

require 'serverspec'

set :backend, :exec

describe 'ci::default' do
  describe command('/usr/bin/java -version') do
    its(:stderr) { should contain('1.7') }
  end

  describe port(8080) do
    it { should be_listening }
  end
  
  describe service('mysql-test') do
    it { should be_enabled }
    it { should be_running }
  end
end

Adding the CI machine in the root Vagrantfile

Vagrantfile

...
  config.vm.define "ci" do |ci|
    ci.vm.hostname = "ci"
    ci.vm.network :private_network, ip: '192.168.33.101'

    ci.vm.provider 'virtualbox' do |vb|
      vb.memory = 1024
    end

    ci.vm.provision :chef_zero, install: true do |chef|
      chef.verbose_logging
      chef.nodes_path = 'cookbooks'
      chef.file_cache_path = '/var/chef/cache'
      chef.add_recipe 'ci::default'
      chef.json = {}
    end
  end
...

And then adds the cookbook ci as dependecy

Berksfile

source 'https://supermarket.chef.io'

cookbook 'sample-app', path: './cookbooks/sample-app'
cookbook 'ci', path: './cookbooks/ci'

run berks update

Adding a Test machine

The test machine is used to simulate a production like environment. We can use to test deploy task or as a staging phase.

The cookbook used to proviion the test machine is the same used for dev environment, with small changes.

cd cookbooks/sample-app

We have to modify the default recipe:

recipes/default.rb

...
['db_notes', 'db_notes_test'].each do |database_name|
  mysql_database database_name do
    connection(
      :host     => '127.0.0.1',
      :username => 'root',
      :password => 'root'
    )
    action :create
  end
end
...

In a test environment we don't need a database used for integration, so we can continue by extracting the list ['db_notes', 'db_notes_test'] as attribute of the cookbook, in order to assign new values programmatically during the provision.

atributes/default.rb

default['java']['jdk_version'] = '7'

default['databases'] = ['db_notes', 'db_notes_test']

recipes/default.rb

...
node['databases'].each do |database_name|
  mysql_database database_name do
    connection(
      :host     => '127.0.0.1',
      :username => 'root',
      :password => 'root'
    )
    action :create
  end
end
...

Now we can add a new test machine in our root Vagrantfile

Vagrantfile

...
  config.vm.define "test" do |test|
    test.vm.hostname = "test"
    test.vm.network :private_network, ip: '192.168.33.102'

    test.vm.provider 'virtualbox' do |vb|
      vb.memory = 1024
    end

    test.vm.provision :chef_zero, install: true do |chef|
      chef.verbose_logging
      chef.nodes_path = 'cookbooks'
      chef.file_cache_path = '/var/chef/cache'
      chef.add_recipe 'sample-app::default'
      chef.json = {
        "databases": ["db_notes"]
      }
    end
  end
...

We are telling the cookbook to use the new attribute databases with only a database. Cool!

run vagrant up test in order to boot the test machine.

We'd like to add a reverse proxy in production

recipes/nginx.rb

package 'nginx'

cookbook_file '/etc/nginx/sites-available/default' do
  source 'nginx-default-site'
  notifies :restart, 'service[nginx]', :delayed
end

service 'nginx'

files/nginx-default-site

upstream backend {
    server localhost:8080;
}

server {
        listen 80 default_server;
        listen [::]:80 default_server ipv6only=on;

        server_name localhost;

        location / {
                proxy_pass http://backend;
        }
}

Now we can add the recipe in the root Vagrantfile

Vagrantfile

...
    chef.add_recipe 'sample-app::default'
    chef.add_recipe 'sample-app::nginx'
...

run vagrant provision test

Automates the application deployment

In order to demostrate how is possible to automates an application deployment we are going to build from scratch a deployment workflow for our application. To do this we use fabric

requirements

Python 2.7
virtualenv

install fabric by pip install -r requirements.txt

then create our first and very simple deploy workflow:

fabfile.py

from fabric.api import *

env.warn_only = True

def deploy():
    stop()
    copy_artefact()
    copy_configuration()
    migrate()
    start()

def copy_artefact():
    put("target/sample-app-1.0-SNAPSHOT.jar", "/home/vagrant/")

def copy_configuration():
    put("configuration.yml", "/home/vagrant/")

def start():
    run("screen -S sample-app -d -m java -jar /home/vagrant/sample-app-1.0-SNAPSHOT.jar server configuration.yml", pty=False)

def stop():
    run("screen -S sample-app -X quit", pty=False)

def migrate():
    run("java -jar /home/vagrant/sample-app-1.0-SNAPSHOT.jar db migrate configuration.yml")

Let's try to deploy the application on the test machine

fab -u vagrant -H 192.168.33.102 deploy

Deployment automation in Jenkins

We want to extend our basic pipeline with a specific task for the deploy. There are some changes we have to introduce in the cookbook sample-app:

cd cookbooks/sample-app

An home and a user used for application deployment

recipes/application_deployment.rb

user 'deployer' do
  shell '/bin/bash'
  home '/home/deployer'
  manage_home true
  action :create
end

directory '/home/deployer/.ssh' do
  owner 'deployer'
  group 'deployer'
  mode '0700'
end

remote_file '/home/deployer/.ssh/authorized_keys' do
  source 'https://gist.githubusercontent.com/joebew42/cfb85d25199b94461c27/raw/ebf41312424286b302d4b7b8f645931d12e0c4b8/deployer.pub'
  owner 'deployer'
  group 'deployer'
  mode '0600'
  action :create
end

Update the root Vagrantfile in order to execute this recipe:

Vagrantfile

...
    test.vm.provision :chef_zero, install: true do |chef|
      chef.verbose_logging
      chef.nodes_path = 'cookbooks'
      chef.file_cache_path = '/var/chef/cache'
      chef.add_recipe 'sample-app::default'
      chef.add_recipe 'sample-app::nginx'
      chef.add_recipe 'sample-app::application_deployment'
      chef.json = {
        "databases": ["db_notes"]
      }
    end
...

Run the provision of the test machine: vagrant provision test

Authorize Jenkins to perform deploy on Test machine

In order to authorize jenkins to perform deploy on test machine we have to change the default recipe:

Install the virtualenv
Install the deployer private key
Add a jenkins plugin to run python code in a virtualenv

cd cookbooks/ci

recipes/default.rb

include_recipe 'java'
include_recipe 'maven'
include_recipe 'jenkins::master'

jenkins_plugin 'git'
jenkins_plugin 'greenballs'
jenkins_plugin 'junit'
jenkins_plugin 'jobConfigHistory'
jenkins_plugin 'delivery-pipeline-plugin'
jenkins_plugin 'shiningpanda' do
  notifies :restart, 'service[jenkins]', :delayed
end

package 'git'
package 'python-virtualenv'
package 'python-dev'

mysql_service 'test' do
  port '3306'
  version '5.5'
  initial_root_password 'root'
  action [:create, :start]
end

mysql2_chef_gem 'default' do
  action [:install]
end

mysql_database 'db_notes_test' do
  connection(
    :host     => '127.0.0.1',
    :username => 'root',
    :password => 'root'
  )
  action :create
end

directory "#{node['jenkins']['master']['home']}/.ssh" do
  owner node['jenkins']['master']['user']
  group node['jenkins']['master']['group']
  mode '0700'
end

remote_file "#{node['jenkins']['master']['home']}/.ssh/deployer" do
  source 'https://gist.githubusercontent.com/joebew42/440c14b70ee305af31f6/raw/2ccd359966d523a026123a434dba262ca9a90e79/deployer'
  owner node['jenkins']['master']['user']
  group node['jenkins']['master']['group']
  mode '0600'
end

Run the provision of the ci machine: vagrant provision ci

Now we can create the job on jenkins to automates the deploy on test machine. We'll adds a simple acceptance test with a curl command.

Deploy in production ? Simple now !

We have already create the test machine that is a production-like machine. We have only to add a new machine in the root Vagrantfile:

Vagrantfile

...
  ['test', 'production'].each_with_index do |environment, index|
    config.vm.define "#{environment}" do |machine|
      machine.vm.hostname = "#{environment}"
      machine.vm.network :private_network, ip: "192.168.33.#{102 + index}"

      machine.vm.provider 'virtualbox' do |vb|
        vb.memory = 1024
      end

      machine.vm.provision :chef_zero, install: true do |chef|
        chef.verbose_logging
        chef.nodes_path = 'cookbooks'
        chef.file_cache_path = '/var/chef/cache'
        chef.add_recipe 'sample-app::default'
        chef.add_recipe 'sample-app::nginx'
        chef.add_recipe 'sample-app::application_deployment'
        chef.json = {
          "databases": ["db_notes"]
        }
      end
    end
  end
...

Run vagrant up production

Logging

We are going to provision an ELK stack (elasticsearch, logstash and kibana)

For practicality (time!) reason a simple logging cookbook can be found here

The complete guide for the logging cookbook can be found here

Add a new machine for logging purpose

Adds the cookbook logging as berks dependency

Berksfile

source 'https://supermarket.chef.io'

cookbook 'sample-app', path: './cookbooks/sample-app'
cookbook 'ci',         path: './cookbooks/ci'
cookbook 'logging',    path: './cookbooks/logging'

Run berks update

Vagrantfile

...
  config.vm.define "management" do |management|
    management.vm.hostname = "management"
    management.vm.network :private_network, ip: '192.168.33.110'

    management.vm.provider 'virtualbox' do |vb|
      vb.memory = 1024
    end

    management.vm.provision :chef_zero, install: true do |chef|
      chef.verbose_logging
      chef.nodes_path = 'cookbooks'
      chef.file_cache_path = '/var/chef/cache'
      chef.add_recipe 'logging::default'
      chef.json = {}
    end
  end
...

Run vagrant up management

Visits http://192.168.33.110:5601 for Kibana Dashboard

Exercise

Take a look at unit and integration tests!

Can you add the same "code coverage" for the ci cookbook ?

Send logs from test/production environment

root Vagrantfile

...
  ['test', 'production'].each_with_index do |environment, index|
    config.vm.define "#{environment}" do |machine|
      machine.vm.hostname = "#{environment}"
      machine.vm.network :private_network, ip: "192.168.33.#{102 + index}"

      machine.vm.provider 'virtualbox' do |vb|
        vb.memory = 1024
      end

      machine.vm.provision :chef_zero, install: true do |chef|
        chef.verbose_logging
        chef.nodes_path = 'cookbooks'
        chef.file_cache_path = '/var/chef/cache'
        chef.add_recipe 'sample-app::default'
        chef.add_recipe 'sample-app::nginx'
        chef.add_recipe 'sample-app::application_deployment'
        chef.add_recipe 'logging::client'
        chef.json = {
          "databases": ["db_notes"],
          "logging": {
              "host": "192.168.33.110"
          }
        }
      end
    end
  end
...

Run vagrant provision test

Monitoring

With Sensu [on-site]

Cloud infrastructure

Create AWS AMI with Packer

Go back to project root and create packer directory

mkdir packer

Define packer template in packer/sample-app.json

{
  "variables": {
    "aws_access_key": "",
    "aws_secret_key": "",
    "name": "default",
    "region": "eu-central-1",
    "source_ami": "ami-7e9b7c11",
    "vpc_id": "",
    "subnet_id": ""
  },

  "builders": [
    {
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "type": "amazon-ebs",
      "region": "{{user `region`}}",
      "source_ami": "{{user `source_ami`}}",
      "ami_virtualization_type": "hvm",
      "vpc_id": "{{user `vpc_id`}}",
      "subnet_id": "{{user `subnet_id`}}",
      "instance_type": "t2.small",
      "ssh_username": "ubuntu",
      "ami_name": "{{user `name`}}-sample-app-{{isotime \"20060102-150405\"}}",
      "tags":  {
        "Name": "{{user `name`}}-sample-app-{{isotime \"20060102-150405\"}}"
      }
    }
  ],

  "provisioners": [
    {
      "type": "chef-solo",
      "cookbook_paths": ["berks-cookbooks"],
      "run_list": [
        "sample-app::default",
        "sample-app::nginx",
        "sample-app::application_deployment"
      ]
    }
  ]
}

Create a Packer a packer/private.json file to customize Packer variables:

{
  "aws_access_key": "{your access key}",
  "aws_secret_key": "{your secret key}",
  "name": "{your name}"
}

Vendorize all needed cookbooks making them availabe to Packer:

berks vendor

Run Packer passing your custom configuration file and the template as arguments:

packer build -var-file=packer/private.json packer/sample-app.json

Take note of generated AMI id.

Create AWS stack with Terraform

Create terraform directory:

mkdir terraform
cd terraform

Define terraform template in terraform/sample-app.tf

provider "aws" {
    access_key = "${var.access_key}"
    secret_key = "${var.secret_key}"
    region = "eu-central-1"
}

resource "aws_security_group" "sample-app" {
    name = "${var.name}-devops-jumpstart-sample-app"
    description = "Security group for web that allows web traffic from internet"

    ingress {
        from_port = 80
        to_port   = 80
        protocol  = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    ingress {
        from_port = 22
        to_port   = 22
        protocol  = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    egress {
        from_port = 0
        to_port = 0
        protocol = "-1"
        cidr_blocks = ["0.0.0.0/0"]
    }
}

resource "aws_instance" "sample-app" {
    instance_type = "t2.small"
    ami = "${var.ami}"
    key_name = "devops-jumpstart"
    security_groups = ["${aws_security_group.sample-app.name}"]

    tags {
      Name = "${var.name} devops-jumpstart sample-app"
    }
}

output "ip" {
    value = "${aws_instance.sample-app.public_ip}"
}

Define terraform variables in terraform/variables.tf

variable "access_key" {}
variable "secret_key" {}
variable "name" {
    default = "user"
}
variable "ami" {}

Define terraform variables values in terraform/terraform.tfvars

access_key = "{your access key}"
secret_key = "{your secret key}"
ami = "{packer generated AMI id}"
name = "{your name}"

Check terraform build plan

terraform plan

Create stack

terraform apply

Show stack state

terraform show

Take note of generated instance IP address. Visit instance IP address in a browser.

Get AWS instance details using aws client

aws configure
aws ec2 describe-instances --filters "Name=tag:Name,Values={user} devops-jumpstart blog"

ivoputzer/devops-workshop.md

Creating a cookbook for our CI server

Requirements

Project setup

Create the ci cookbook

Our first recipe

Fix our recipe first

How to verify that Java is installed ?

Put all things together

Jenkins cookbook

Just one simple integration test

Complete the recipe

Adding the CI machine in the root Vagrantfile

Adding a Test machine

We'd like to add a reverse proxy in production

Automates the application deployment

Deployment automation in Jenkins

An home and a user used for application deployment

Authorize Jenkins to perform deploy on Test machine

Deploy in production ? Simple now !

Logging

Add a new machine for logging purpose

Send logs from test/production environment

Monitoring

Cloud infrastructure

Create AWS AMI with Packer

Create AWS stack with Terraform