jenkins.md

layout	title	author
post	Multi-service CD pipeline with Jenkins	Tor Ivry

Intro

Micro services are awesome! we love them for many reasons - including:

They are small and simple
They are modular and extensibile
We have the freedom to select the best tool for the job

However, this approach also has it's pitfalls.

One of the difficulties is caused by the fact that we use a zoo of technologies and languages. Each of the environments requires a way to do:

package management
compilation (for static languages)
data migrations
testing
deployment

(and also monitoring, aggregated logging, communication protocols and many more).

Goal

A centralized solution for building, testing and deploying our services independently of programming language and framework.

Enter Jenkins

Jenkins is a well known continuous integration server, built in Java.

Find out more by visiting the project on GitHub.
Jenkins is highly customizable, but requires some level of getting your hands dirty.
We have customized Jenkins to create a pipeline for each service. In a typical service pipeline we will have 3 stages:

Build
Deploy to integration
Deploy to production

##Setup An important notice is that the Jenkins server should have a superset of all the environments and databases that we use to build and test out different services.

Example: You have the following services:

Mosby - A node.js service using redis and mongoDB
Stinson - A clojure service using PostgreSQL and redis

Lets also say you use graphite for monitoring and logstash for centralized logging

####Frameworks You will need to have JVM and leiningen for clojure. You will also need to have node and npm installed on the machine.

####Databases, monitoring and centralized logging In your case - redis, PostgreSQL, mongoDB, graphite and logstash.
For each solution you can decide on one of the above:

Install it on the machine
Use service configuration for remote database / service url.

Chef

We use Chef for configuration management, so we simply generated a role containing all the required cookbooks for frameworks and tools we use in our services.

We have added the jenkins cookbook to the role and viola!
We are now ready to start customizing our Jenkins solution.

1. Build

This is the first stage of the service pipeline. In this stage we will:

Install package dependencies,
Compile the code when the language is statically typed
Run data migrations
Add custom installation scripts (e.g. seed the database)
Run the tests

Build job template

Add a new job and use the following settings:

Project name: build_mosby

This is the job name, used as identifier. It will also be used to set the job URL and will be used throughout the Jenkins API.
GitHub project: https://github.com/ted/mosby

We maintain each service in a separate GitHub repository.

Embeddable-Build-Status makes it simple to show the current build status in your repository README

GitHub Plugin let's you link your repository code to the build job. You also need this plugin to trigger a build on every push to the repository.
Source Code Management
- Git
  - Repository URL: [email protected]:ted/mosby.git
    
    This is the URL used for the git clone command performed by Jenkins
Build Triggers
- Build when a change is pushed to GitHub
  
  Every push to the remote repository triggers a job run.
Build
- Excecute shell
  - Command: ./build
  This is where we do the actual magic. See below
Post-build Actions
- HipChat Notifications
  
  Jenkins HipChat Plugin Let's you get HipChat notifications for jenkins failures / successful job runs
- Publish JUnit test result report
  - Test report XML: <relative path to the reports directory>
    
    This is an optional post build action that allows you to display test stats on Jenkins dashboard

The result will be saved in your job directory and can be cloned as a baseline create the rest of your build jobs. You can see the xml here.

Typical build script

####ruby:

If you manage ruby versions with rbenv you can use the rbenv plugin and setup ruby version in the Build Environment section of the job configuration.

Jenkins ruby metrics Plugin allows you to publish Rcov metrics as part of the job stats

See ci_reporter for test reports in JUnit xml format required by jenkins. For tests coverage see simplecov and simplecov-rcov

rails with rspec and cucumber:

bundle install
rake db:create db:migrate db:test:prepare
bundle exec rake cucumber
RAILS_ENV=test bundle exec rake ci:setup:rspec spec

Regular ruby process / sinatra with minitest:
```
bundle install
bundle exec rake test
```

####golang:

We are using gom to manage our golang package dependencies.

export PATH=$PATH:$HOME/go/bin
gom install
gom test
GOOS=linux GOARCH=amd64 gom build

####node.js:

npm install
./node_modules/.bin/mocha spec/**/*.js

2. Deploy to integration

We use an integration server that runs our services with their most up to date versions.
We use it to test for new features and the interactions between multiple services before we send them to production

This job is very simple.
It runs a deployment of the service to the integration server and runs a sanity to validate that the deploy was successful.

Project name: deploy_mosby_integration

The suffix denotes the target environment
Build Triggers
- Build after other projects are built
  - Project names: build_<service name>
    
    This is where the pipline magic happens. This job executes only run when the build_mosby job ends successfuly.
Build
- Excecute shell
  - Command: ./deploy
    
    Similar to what we did in the build job.
    Depending on the service, we can use Capistrano, Fabric, gradle or even a simple upload to s3 inside the deploy script.

3. Deploy to production

Similar to Deploy to integration with few nuances:

Triggered by the successful run of deploy_mosby_integration
We use hubot command to run the job instead of triggers for critical services.

The deploy should be something like:

servers_we_want_to_deploy_to.each do |server|
  load_balancer.remove server 
  deploy_to server 
  sanity_of server 
  load_balancer.add server 
end

##Dashboard

You can customize the dashboard view to display the desired stats such as coverage and test results graphs.
Build Monitor View can be used on a large screen to get something like this:

Summary

We have shown how to create a Jenkins setup that provides the abillity to add a new service CD pipeline in just a few minutes.
We have achieved this simplicity by using job templates for build and deploy.

torkale/jenkins.md