Chef Local Development Workflow with ChefDK, Vagrant, VirtualBox, and Test Kitchen

Overview

The following document is intended to be a quick guide to getting you setup for doing local development with Chef. This guide was created on my MacBook, but should work fine with Linux, and Windows workstations as well.

Quick review on fundamental tenets of Chef

Workstation - A workstation is a computer that is configured to run various Chef command-line tools that synchronize with a chef-repo, author cookbooks, interact with the Chef server, interact with nodes, or applications like Chef Delivery
Node - A node is any machine—physical, virtual, cloud, network device, etc.—that is under management by Chef.
Chef Server- The Chef server acts as a hub for configuration data. The Chef server stores cookbooks, the policies that are applied to nodes, and metadata that describes each registered node that is being managed by the chef-client.
Resources - The fundamental building block of chef describing something you want to configure like a package, template, or service, and the state that it is in
Recipes - An ordered list of resources
Cookbooks - A collection of resources and their supporting files. Typically map 1:1 to something you want to configure (i.e. mysql_cookbook, httpd_cookbook, sensu_cookbook)
Environments - Environments can be created to reflect each organization’s patterns and workflow. For example, creating production, staging, testing, and development environments.
Roles - A role is a way to define certain patterns and processes that exist across nodes in an organization as belonging to a single job function. Each role consists of zero (or more) attributes and a run-list. Each node can have zero (or more) roles assigned to it. When a role is run against a node, the configuration details of that node are compared against the attributes of the role, and then the contents of that role’s run-list are applied to the node’s configuration details. When a chef-client runs, it merges its own attributes and run-lists with those contained within each assigned role.
Data Bags - A data bag is a global variable that is stored as JSON data and is accessible from a Chef server. A data bag is indexed for searching and can be loaded by a recipe or accessed during a search.

Setup

In order to get going you are going to need to install the following software on your local workstation.

Chef Development Kit (ChefDK)
Vagrant
VirtualBox
Programmer's Text editor like Atom, Sublime Text 2, or Notepad++

Once installed validate your installation by running the following commands*

Validate ChefDK Install

$ chef -v
Chef Development Kit Version: 0.15.16
chef-client version: 12.11.18
delivery version: master (444effdf9c81908795e88157f01cd667a6c43b5f)
berks version: 4.3.5
kitchen version: 1.10.0

Validate Vagrant Install

$ vagrant -v
Vagrant 1.8.4

Validate VirtualBox Installation

$ VBoxManage -v
5.0.24r108355

Validation steps were tested on MacOS. Commands may not work on Windows and Linux.

A first cookbook

If you are new to Chef or software development for that matter it is probably a good idea to create a working directory for all of your source code. I use a folder called src in my home directory (i.e. /Users/scottford/src).

Create an http cookbook

Open up the command terminal on your workstation (Powershell, Cmd, Terminal, iTerm, Bash, etc), change directory into your source directory and run the following command:

$ chef generate cookbook <COMPANY_NAME>_httpd

NOTE: I created a chef_httpd cookbook so my examples from here on out will reference that cookbook name

Change directory into the cookbook

$ cd chef_httpd

Open the cookbook in your editor

Open your text editor and open the folder for the cookbook you created. Doing this will allow you to see the entire contents of the httpd cookbook.

Intro to Test Kitchen

Test Kitchen is a test harness tool to execute your configured code on one or more platforms in isolation. A driver plugin architecture is used which lets you run your code on various cloud providers and virtualization technologies such as Amazon EC2, Blue Box, CloudStack, Digital Ocean, Rackspace, OpenStack, Vagrant, Docker, LXC containers, and more. Many testing frameworks are already supported out of the box including Inspec, Bats, shUnit2, RSpec, Serverspec, with others being created weekly.

Introduction to the `.kitchen.yml` configuration file

The .kitchen.yml file found inside the cookbook folder we just created when we ran chef generate cookbook is the configuration file for Test Kitchen. It is here where you can set the driver you want to use, what platforms you need to support, testing frameworks, as well as the Chef run_list you want Test Kitchen to run.

The first thing you should note is that the file is .yml, which can be a little finicky if you have not worked with it before. You must NOT USE TABS, spaces only!

Very briefly we can cover the 4 main sections you're likely to find in a .kitchen.yml file:

driver: This is where we configure the behaviour of the Kitchen Driver - the component that is responsible for creating a machine that we'll use to test our cookbook. Here we set up basics like credentials, ssh usernames, sudo requirements, etc. Each Driver is responsible for requiring and using the configuration here. Under this section we have driver.name: This tells Test Kitchen that we want to use the kitchen-vagrant driver by default unless otherwise specified.
provisioner: This tells Test Kitchen how to run Chef, to apply the code in our cookbook to the machine under test. The default provisioner is chef_zero which creates a small in memory Chef Server.
platforms: This is a list of operation systems on which we want to run our code. Note that the operating system's version, architecture, cloud environment, etc. might be relevant to what Test Kitchen considers a Platform.
suites: This section defines what we want to test. It includes the Chef run-list and any node attribute setups that we want run on each Platform above. For example, we might want to test the MySQL client cookbook code separately from the server cookbook code for maximum isolation.

Edit `.kitchen.yml`

Copy/Paste the following into your .kitchen.yml

---
driver:
  name: vagrant

 ## The private_network feature lets you setup a private network on the VM guest
 ## via localhost on the host.
 ## see also: https://www.vagrantup.com/docs/networking/private_network.html

 network:
   - ["private_network", {ip: "33.33.33.33"}]

provisioner:
  name: chef_zero

## The verifier section determines which test platform you want to use.
verifier:
  name: inspec
  format: doc

platforms:
  - name: centos-6.7

suites:
  - name: default
    run_list:
      - recipe[chef_httpd::default]
    attributes:

List Out Configured VMs

Running the command kitchen list lists out your configuration. It also validates the file. If you get any errors you most likely have a formatting error. Try copying/pasting from the example above.

$ kitchen list
Instance           Driver   Provisioner  Verifier  Transport  Last Action
default-centos-67  Vagrant  ChefZero     Inspec    Ssh        <Not Created>

Bring up a local VM

The kitchen create command launches a VM using the driver configured. In our example we are using Vagrant to launch a VM locally. The first time you launch a specified platform you will need to pull down the Vagrant box file locally which is stored in ~/.vagrant.d/boxes. Each subsequent launch of that platform will use the locally stored file.

$ kitchen create
-----> Starting Kitchen (v1.10.0)
-----> Creating <default-centos-67>...
       Bringing machine 'default' up with 'virtualbox' provider...
       ==> default: Importing base box 'bento/centos-6.7'...
==> default: Matching MAC address for NAT networking...
       ==> default: Checking if box 'bento/centos-6.7' is up to date...
       ==> default: A newer version of the box 'bento/centos-6.7' is available! You currently
       ==> default: have version '2.2.5'. The latest is version '2.2.7'. Run
       ==> default: `vagrant box update` to update.
       ==> default: Setting the name of the VM: kitchen-chef_httpd-default-centos-67_default_1467922394968_13785
...
-----> Kitchen is finished. (0m37.77s)

View the new status of the running VM with `kitchen list`

Running kitchen list now shows the Last Action as Created.

$ kitchen list
Instance           Driver   Provisioner  Verifier  Transport  Last Action
default-centos-67  Vagrant  ChefZero     Inspec    Ssh        Created

Login to your guest instance with `kitchen login`

You now can actually login to your running VM with the command kitchen login and execute commands just as you would on any other RHEL/CentOS system.

$ kitchen login
Last login: Thu Jul  7 20:13:57 2016 from 10.0.2.2
[vagrant@default-centos-67 ~]$ cat /etc/redhat-release
CentOS release 6.7 (Final)
[vagrant@default-centos-67 ~]$ exit
logout
Connection to 127.0.0.1 closed.

Run `kitchen converge`

Running kitchen converge checks to see that the box has been created, installs Chef on the target platform, and then runs chef-client runs the run_list for the suite.

$ kitchen converge
-----> Starting Kitchen (v1.10.0)
-----> Converging <default-centos-67>...
       Preparing files for transfer
       Preparing dna.json
       Resolving cookbook dependencies with Berkshelf 4.3.5...
       Removing non-cookbook files before transfer
       Preparing validation.pem
       Preparing client.rb
-----> Installing Chef Omnibus (install only if missing)
       Downloading https://omnitruck.chef.io/install.sh to file /tmp/install.sh
       Trying wget...
       Download complete.
       el 6 x86_64
       Getting information for chef stable  for el...
       downloading https://omnitruck.chef.io/stable/chef/metadata?v=&p=el&pv=6&m=x86_64
         to file /tmp/install.sh.3080/metadata.txt
       ...

       Installing chef
       installing with rpm...
       ...
       Thank you for installing Chef!
       Transferring files to <default-centos-67>
       Starting Chef Client, version 12.12.13
       Creating a new client identity for default-centos-67 using the validator key.
       resolving cookbooks for run list: ["chef_httpd::default"]
       Synchronizing Cookbooks:
         - chef_httpd (0.1.0)
       Installing Cookbook Gems:
       Compiling Cookbooks...
       Converging 0 resources

       Running handlers:
       Running handlers complete
       Chef Client finished, 0/0 resources updated in 02 seconds
       Finished converging <default-centos-67> (0m22.80s).
-----> Kitchen is finished. (0m25.76s)

View the new status of the running VM with `kitchen list`

Running kitchen list now shows the Last Action as Converged

$ kitchen list
Instance           Driver   Provisioner  Verifier  Transport  Last Action
default-centos-67  Vagrant  ChefZero     Inspec    Ssh        Converged

Writing and Testing a default recipe

Edit the default recipe to have Chef install the `httpd` package, and start the `httpd` service.

Open up recipes\default.rb

#
# Cookbook Name:: chef_httpd
# Recipe:: default
#
# Copyright (c) 2015 The Authors, All Rights Reserved.

package 'httpd' do
  action :install
end

service 'httpd' do
  action [:start, :enable]
end

Re-run Chef to converge the recipe

$ kitchen converge
-----> Starting Kitchen (v1.4.2)
-----> Converging <default-centos-71>...
       ...
       Synchronizing Cookbooks:
         - chef_httpd (0.1.0)
       Installing Cookbook Gems:
       Compiling Cookbooks...
       Converging 2 resources
       Recipe: chef_httpd::default
         * yum_package[httpd] action install
           - install version 2.2.15-53.el6.centos of package httpd
         * service[httpd] action enable
           - enable service service[httpd]
         * service[httpd] action start
           - start service service[httpd]

       Running handlers:
       Running handlers complete
       Chef Client finished, 3/3 resources updated in 10 seconds
       Finished converging <default-centos-67> (0m21.02s).

Test your webserver manually

Open up a browser and go to http://33.33.33.33 to see the default configuration page for the Apache Webserver.

Automated testing with Inspec

While it is nice that we can validate our webserver manually, wouldn't it be better to do our validation in an automated fashion? We can!

We are going to use a testing framework called Inspec (Infrastructure Specification) to test the configuration of our webserver.

Install the latest Inspec

$ chef gem install inspec
Successfully installed inspec-0.26.0
1 gem installed

Create a directory to store your Inspec tests

Open up your terminal again and change directory into your cookbook. Create a directory to store your your Inspec tests.

$ mkdir -p test/integration/default/inspec

Write your first test

In your editor, create a new file test/integration/default/inspec/default_spec.rb and add the following test

describe port(80) do
  it { should be_listening }
end

Run the integration test with `kitchen verify`

$ kitchen verify
-----> Starting Kitchen (v1.10.0)
-----> Setting up <default-centos-67>...
       Finished setting up <default-centos-67> (0m0.00s).
-----> Verifying <default-centos-67>...
       Detected alternative framework tests for `inspec`
       Detected alternative framework tests for `serverspec`
       Use `/Users/scottford/src/chef_httpd/test/integration/default/inspec` for testing

Port 80
  should be listening

Finished in 0.12748 seconds (files took 1.18 seconds to load)
1 example, 0 failures

       Finished verifying <default-centos-67> (0m1.01s).
-----> Kitchen is finished. (0m3.00s)

Destroy the running VM

$ kitchen destroy

larryebaum/chef_local_development_workflow.md

Chef Local Development Workflow with ChefDK, Vagrant, VirtualBox, and Test Kitchen

Overview

Quick review on fundamental tenets of Chef

Setup

Validate ChefDK Install

Validate Vagrant Install

Validate VirtualBox Installation

A first cookbook

Create an http cookbook

Change directory into the cookbook

Open the cookbook in your editor

Intro to Test Kitchen

Introduction to the .kitchen.yml configuration file

Edit .kitchen.yml

List Out Configured VMs

Bring up a local VM

View the new status of the running VM with kitchen list

Login to your guest instance with kitchen login

Run kitchen converge

View the new status of the running VM with kitchen list

Writing and Testing a default recipe

Edit the default recipe to have Chef install the httpd package, and start the httpd service.

Re-run Chef to converge the recipe

Test your webserver manually

Automated testing with Inspec

Install the latest Inspec

Create a directory to store your Inspec tests

Write your first test

Run the integration test with kitchen verify

Destroy the running VM

Introduction to the `.kitchen.yml` configuration file

Edit `.kitchen.yml`

View the new status of the running VM with `kitchen list`

Login to your guest instance with `kitchen login`

Run `kitchen converge`

View the new status of the running VM with `kitchen list`

Edit the default recipe to have Chef install the `httpd` package, and start the `httpd` service.

Run the integration test with `kitchen verify`