Introduction to Vagrant and Chef

Preamble

The primary purpose of the tutorial is to introduce Vagrant and Chef, however, there are a number of other technologies and concepts that will be used along the way. Links are provided to the some of these technologies from which more information can be found. You are encouraged to go to the relevant documentation for a better and deeper understanding. For commands that are used in the shell the man pages can be used to get more information about the command.

Instructions for commands assume a Unix-like (Linux/MacOS) environment. Commands will be in a code block. Example:

    $ mkdir new_directory

means make a directory called new_directory the $ and everything that precedes it are not to be typed.

If a directory path precedes the $ then that is the directory from which the command should be run So /home/user/tutorial $ ls

means run the command ls in the directory /home/user/tutorial

When used in the context of file paths, a ~ at the start of the path indicates a directory relative to the user's home folders. Therefore, ~/tutorial is a the path /home/user/tutorial.

Example:

    $ man man

shows the manual for the man command.

Notes and instructions will be written in the following form

Note: This is a note. Read this.

What is Vagrant?

Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team. - vagrantup.com

Imagine you would like to develop and test a web application, you could create the required vms, install the OSes, install and configure the required software and then proceed to start your development. But what happens when you need to repeat the process? You would have to go through the tedium of setting up everything again. Sure you could write your own custom scripts, but then more than likely you would have to edit them for a new project.

This is where Vagrant steps in. Vagrant allows you to create your custom environments and be able to tear them down, reconfigure and test using its framework. This frees you from the tedium of these tasks and allows you to proceed through your dev-test cycles quickly.

Task 1 - Minimal Vagrant setup

Follow the instructions to install Vagrant for your OS.

Once that is done, create a new directory in which you will do this tutorial.

Note: The directory name and location can be changed.

    ~ $ mkdir vagrant_chef_tutorial
    ~ $ cd vagrant_chef_tutorial

Vagrantfile

The primary function of the Vagrantfile is to describe the type of machine required for a project, and how to configure and provision these machines. Vagrantfiles are called Vagrantfiles because the actual literal filename for the file is Vagrantfile (casing doesnt matter unless your file system is running in a strict case sensitive mode). - vagrantup.com

Create the skeleton Vagrant file

    ~/vagrant_chef_tutorial $ vagrant init

Use your text editor to open the resulting file Vagrantfile. Note: A text editor with syntax highlighting for the Ruby language will make the file more readable. The Vagrantfile by default contains more comments than commands. The comments give insight on the possible commands that can be included in the creation of your virtual machine and hence your dev environment.

In the file you should see these lines

    # Every Vagrant development environment requires a box. You can search for
    # boxes at https://atlas.hashicorp.com/search.
    config.vm.box = "base"

This introduces the next concept, boxes.

Boxes

Boxes are the package format for Vagrant environments. A box can be used by anyone on any platform that Vagrant supports to bring up an identical working environment. - vagrantup.com

Simply the vagrant box is an image of an OS in which custom applications can be installed. Boxes can be found hashicorp and vagrantbox. For this tutorial we will be using a CentOS box.

Install the box

    vagrant box add centos6 https://s3.amazonaws.com/itmat-public/centos-6.3-chef-10.14.2.box

This command may take some time depending on your Internet connection. It will download the box and register it with vagrant under the name centos6.

Once this command has completed we can edit the Vagrantfile. Change the line

    config.vm.box = "base"

    config.vm.box = "centos6"

and save the file.

Now run

    ~/vagrant_chef_tutorial $ vagrant up

At this point you should see something happening. Vagrant is going to bring up a vm based on the box that you specified.

Sample output:

    ==> default: Importing base box 'chef-cent6'...
    ==> default: Matching MAC address for NAT networking...
    ==> default: Setting the name of the VM: tutorial_default_1448576155809_78945
    ==> default: Clearing any previously set network interfaces...
    ==> default: Preparing network interfaces based on configuration...
        default: Adapter 1: nat
    ==> default: Forwarding ports...
        default: 22 => 2222 (adapter 1)
    ==> default: Booting VM...
    ==> default: Waiting for machine to boot. This may take a few minutes...
        default: SSH address: 127.0.0.1:2222
        default: SSH username: vagrant
        default: SSH auth method: private key
        default: Warning: Connection timeout. Retrying...
        default: 
        default: Vagrant insecure key detected. Vagrant will automatically replace
        default: this with a newly generated keypair for better security.
        default: 
        default: Inserting generated public key within guest...
        default: Removing insecure key from the guest if it's present...
        default: Key inserted! Disconnecting and reconnecting using new SSH key...
    ==> default: Machine booted and ready!
    ==> default: Checking for guest additions in VM...
        default: The guest additions on this VM do not match the installed version of
        default: VirtualBox! In most cases this is fine, but in rare cases it can
        default: prevent things such as shared folders from working properly. If you see
        default: shared folder errors, please make sure the guest additions within the
        default: virtual machine match the version of VirtualBox you have installed on
        default: your host and reload your VM.
        default: 
        default: Guest Additions Version: 4.1.22
        default: VirtualBox Version: 4.3
    ==> default: Mounting shared folders...
        default: /vagrant => /tmp/tutorial

Let's play around a little. We are going to login via ssh to the newly created machine.

    ~/vagrant_chef_tutorial $ vagrant ssh

The prompt should now look like

    [vagrant@localhost ~]$

indicating that you are now in the machine that vagrant just brought up.

    [vagrant@localhost ~]$ ls 
    [vagrant@localhost ~]$ cd /
    [vagrant@localhost ~]$ ls
    [vagrant@localhost ~]$ cd /tmp
    [vagrant@localhost ~]$ touch file
    [vagrant@localhost ~]$ ls -l file

All these commands are being executed in the virtual machine.

    [vagrant@localhost ~]$ exit

Now that we have a running machine we can now proceed to developing our chef cookbook.

Chef

Chef is a company & configuration management tool written in Ruby and Erlang. It uses a pure-Ruby, domain-specific language (DSL) for writing system configuration "recipes". Chef is used to streamline the task of configuring and maintaining a company's servers ... - Wikipedia

While vagrant helps configure virtual machines and ensures that their configurations are consistent, Chef is used to ensure that the state of the machine is consistent. To achieve this users create "recipes" which programmatically specify the state that the system should be in. Therefore, installing the specific version of a software, ensuring that configuration files are present and have the relevant values all fall withiin the broad scope of Chef.

Chef is usually used in client-server setup. However, it provides the option to be run stand-alone. This option is provided by Vagrant and hence it is what we will use.

The overview of Chef explains the main components of Chef. For this tutorial we will focus on two

Cookbooks

A cookbook is the fundamental unit of configuration and policy distribution. A cookbook defines a scenario and contains everything that is required to support that scenario

Recipes

A recipe is the most fundamental configuration element within the organization. A recipe ... must define everything that is required to configure part of a system.

Recipes use Resources and Providers to A resource is a statement of configuration policy that:

Describes the desired state for a configuration item
Declares the steps needed to bring that item to the desired state
Specifies a resource type—such as package, template, or service
Lists additional details (also known as resource properties), as necessary
Are grouped into recipes, which describe working configurations

Chef version

Let's update the version of chef in the virtual machine

vagrant_chef_tutorial $ vagrant up

If the machine is not running this will bring it up, if it is up then vagrant will let us know.

    vagrant_chef_tutorial $ vagrant ssh
    [vagrant@localhost ~]$ sudo su
    [vagrant@localhost ~]# curl -sL https://www.chef.io/chef/install.sh | bash -s -- -v "11.12" 
    [vagrant@localhost ~]# exit
    vagrant_chef_tutorial $ exit

The version of Chef in the vm is now the one we want.

Simple Recipe

To begin we will create a directory on our machine and in it put a file called hello_world.txt.

    vagrant_chef_tutorial $ mkdir -p cookbooks/hello-world
    vagrant_chef_tutorial $ cd cookbooks/hello-world
    vagrant_chef_tutorial $ mkdir attributes  definitions  files  libraries  providers  recipes  resources  templates
    vagrant_chef_tutorial $ touch  metadata.rb README.md

You have just created the basic directory structure for a cookbook. In metadata.rb add the following lines

    name             'hello-world'
    maintainer       ''
    maintainer_email ''
    license          'All rights reserved'
    description      'Installs/Configures hello-world'
    long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
    version          '0.0.1'

This is basic information needed by Chef for the cookbook.

In the recipes folder create a file called default.rb. This is the default recipe for the cookbook and is used by Chef when the cookbook is run if no other recipe is specified. We will now use the directory resource to create a directory.

Add in default.rb

directory '/home/vagrant/hello-world' do
    owner 'vagrant'
    group 'vagrant'
    mode  '0755'
    action :create
end

This code block will create the directory hello-world with the owner and group being the vagrant user.

File System Modes explains the mode line and the permissions granted

At this point the virtual machine should be running. To check the status

    vagrant_chef_tutorial $ vagrant status

The output should show running. If the machine is not running execute vagrant up to start the machine.

The machine is running and we want to provision it using the newly created recipe. To do this we need to update the Vagrantfile Add this section in your Vagrantfile

config.vm.provision :chef_solo do |chef|
    chef.install = false
    chef.cookbooks_path = "cookbooks"
end

This tells vagrant the directory in which to look for our chef recipes. Now we want to tell chef to use our recipe Edit the file

    config.vm.provision :chef_solo do |chef|
        chef.cookbooks_path = "cookbooks"
        chef.log_level = "debug"

        chef.add_recipe  "hello-world"
    end

Now we should be able to provision the machine

    vagrant_chef_tutorial $ vagrant provision

This should run chef-solo and it should run successfully (no red output lines). Enter the vm and check for the directory. Hint use the ls command. Once you have satisfied that the directory exists, exit the vm.

Tutorial Recipe

We want to create a recipe which will install and configure the Nginx server and configure it to serve a page.

Nginx Installation

Create a new cookbook folder website1 with the same structure as before. Edit the metadata.rb file to reflect the new cookbook information. We need to get the latest version of Nginx, to do this we will use the repo package provided by Nginx and then install the actual program. We will tell Chef to do all of this for us.

Note: There are usually more than one ways of implementing these workflows, the important thing is to achieve idempotence(more here and here). It is advised to use the resources and providers as much as is possible.

Let use get and install the rpm package from the nginx site. We will use the remote_file and yum_package resources to do this. In your default.rb put

remote_file '/tmp/nginx-release-centos-6-0.el6.ngx.noarch.rpm' do 
    source 'http://nginx.org/packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm'
    path 
    action :create_if_missing
end

Note: the rpm link was retrieved from the nginx linux packages for CentOS 6 since that is the OS version in our vm.

The create_if_missing flag is to skip retrieving the file once it has been downloaded.

Now for the installation

Add each of the following blocks of code to default.rb and run vagrant provision to see the changes taking place.

This is an example of how you could create incrementally and debug/confirm that your recipe is working. These steps could all be written at the same time and then tested. It's all up to the implementer

yum_package 'nginx repo' do 
    source '/tmp/nginx-release-centos-6-0.el6.ngx.noarch.rpm'
    action :install
end

At this point we can install the nginx web server.

yum_package 'nginx' do 
    action :install
end
    ```
We now have nginx installed and ready to be configured. By default the nginx service is not started so we will start it.

```ruby
service 'nginx' do 
    action :start
end

Port forwarding

At this point if this was our local machine we would be able to browse to http://localhost and see that nginx serving its default page. However since we are running nginx in a vm we can't browse to that location. To get around this Vagrant allows us to forward a port from our host machine to our virtual machine. To do this we need to edit our Vagrantfile.

If you are using the default Vagrantfile then this line

# config.vm.network "forwarded_port", guest: 80, host: 8080

should be present. Simply uncomment the line (remove the leading #), save the file and run

    vagrant_chef_tutorial $ vagrant reload

Now visit http://localhost:8080 in your browser.

Ordinarily you would expect a page to load indicating that everything is working properly. However, this does not happen. The reason is that the firewall running on the OS is not configured to allow traffic over port 80 (on which the webserver is listening). The proper response in this situation would be to configure the firewall rules to allow traffic on port 80. However, for the sake of focus we will not be doing this here. We will simply stop the firewall service as part of our provisioning (an ugly hack that should not be done). I leave editing the firewall rules, temporarily and/or persistently, up to the reader as an exercise. The firewall service is iptables so to our recipe we can add

service 'iptables' do
    action :stop
    ignore_failure true
end

run vagrant provision and the revisit http://localhost:8080. Now you should see the Nginx welcome page.

Custom Page

We are going to create a custom page in a custom location. To do this we will need to

create a custom config file in /etc/nginx/conf.d
add a custom location for our page
add our page in our custom location
restart the web server to load our changes

File creation can be done using the file or template resource.

Use the file resource to manage files directly on a node.

A cookbook template is an Embedded Ruby (ERB) template that is used to dynamically generate static text files.

We will use both methods here. There is no dynamism in our template but it is here as an example of using templates.

Run the following commands

    cookbooks/website/ $ mkdir templates/default
    cookbooks/website/ $ touch templates/default/tutorial.conf.erb

The custom config file is templated here. Config files are usually good candidates for templates since there may be values that need to be changed when dealing with multiple environments.

Edit tutorial.conf.erb

See Nginx documentation for the syntax explanation.

    server {
        listen 81;
        server_name localhost;

        location / {
            root /var/mywww; 
            index tutorial.html;
        }
    }

We can now use the template in our recipe.

The following snippet should be added to default.rb.

Note: the creation of the directory /var/mywww/ is not included and should be done by the reader

# create custom page location 
## Implement Here

# put HTML file in custom location
file '/var/mywww/tutorial.html' do
    content '<html>
                <head><title>Tutorial</title></head>
                <body>
                    <h2> Hello World !!! </h2>
                    <h3> Cooking Good !!! </h3>
                </body>
             </html>'
end

template '/etc/nginx/conf.d/tutorial.conf' do
    source 'tutorial.conf.erb'
    action :create
end

Note: The configuration for the server in tutorial.conf is set to listen on port 81. You will need to forward a port from the host to access this port.

Once you have finished the chef recipe and provisioned the machine you should be able to view the page at http://localhost:__forwarded_host_port__.

Final Test

Recall that the point of writing the chef recipes is to have configuration scripts that allow fresh machines to be put in a known good state. To check that your recipe can achieve this, we will destroy our machine and try to provision it again.

    vagrant_chef_tutorial $ vagrant destroy
    vagrant_chef_tutorial $ vagrant up

This should not work. Can you tell why?

The reason is the version of Chef on the machine by default. Install the correct version as done before and then run

    vagrant_chef_tutorial $ vagrant provision

If you have written your script correctly then http://localhost:__forwarded_host_port__ should show you the tutorial page.

Note: Installing the new version of Chef as we have done it here is hacky and not in line with properly provisioning the machine. Vagrant provides the shell provisioner which could be used to rectify this. This is left as an exercise to the reader.

Conclusion

This is only the beginning. Chef, Vagrant, the shell and all the tools shown here have many more options available. If you are so motivated you can continue to modify, edit and install anything else in your new machine. Follow the links, search and explore.

kahbenya/intro_vagrant_chef.md