Jenkins: Build only when necessary

jenkins_build_skip.md

Build Only When Necessary

In testing environments, it is not always useful repeating tasks which can cost developers valuable time. Within a project its safe to assume that not all files are changed, so why rebuild?

This tutorial outlines a method to reduce unnecessarly build steps. Use your best judgement of course. It might be useful to add in a build on occassion to ensure third party dependencies are available and will not disrupt a production deployment.

Jenkins Setup

The git plugin in Jenkins provides several useful environment variables which can be used to avoid build tasks. Make sure you setup the git plugin before going any further.

Now, lets jump into the code.

Script: jenkins/scripts/list_files_changed.sh

#!/bin/bash

set -e

if [[ "$1" == 'help' ]]; then
    echo "Usage: $0 [git previous commit]"
    echo
    echo 'This Jenkins script will print all files that have changed between'
    echo 'two commits. Default is `GIT_PREVIOUS_SUCCESSFUL_COMMIT` and `GIT_COMMIT`, but'
    echo 'can be over-written to any valid git commit hash.'
    echo
    echo 'Note this is to be called from a Jenkins environment with the git'
    echo 'plugin installed. Please see available environment variables.'

    exit 0
fi

if [[ $DEBUG -ne 0 ]]; then
    echo "DEBUG: GIT_COMMIT=${GIT_COMMIT}"
    echo "DEBUG: GIT_PREVIOUS_COMMIT=${GIT_PREVIOUS_COMMIT}"
    echo "DEBUG: GIT_PREVIOUS_SUCCESSFUL_COMMIT=${GIT_PREVIOUS_SUCCESSFUL_COMMIT}"
fi

if [[ -z "${GIT_COMMIT}" ]]; then
    echo 'ERROR: `GIT_COMMIT` was not defined!'
    exit 1
fi

current_commit="${GIT_COMMIT}"
user_override_previous_commit="$1"
previous_commit="${user_override_previous_commit:=$GIT_PREVIOUS_SUCCESSFUL_COMMIT}"

if [[ -z "${previous_commit}" ]]; then
    echo 'ERROR: Either `GIT_PREVIOUS_SUCCESSFUL_COMMIT` was not defined, or user did not'
    echo 'override with previous commit.'
    exit 2
fi

if [[ $DEBUG -ne 0 ]]; then
    echo "DEBUG: Comparing git commit '${previous_commit}' to '${current_commit}' ..."
fi

git --no-pager diff --name-only ${previous_commit} ${current_commit}

Simply, when run in Jenkins environment, by default the script will use the GIT_COMMIT and GIT_PREVIOUS_SUCCESSFUL_COMMIT to determine which files have changed between these commits. A developer can use this script to test if a file has been updated since the last successful build. This will ensure that any code taking advantage of this script will retry until the subtask completes.

If it isn't clear yet, hopefully a code example will help:

if [ -z "$(sh jenkins/scripts/list_files_changed.sh | grep 'packer-ami.json')" ]; then
    echo 'Skipping: No changes detected since last successful build.'
else
    echo 'Run task: detected changes since last successful build.'
    # ...
fi

Jenkins: Forwarding Environment Variables to Other Jobs

The parent job may not pass along git plugin environment variables by default, you may have to manually configure Jenkins to pass in GIT_COMMIT and GIT_PREVIOUS_SUCCESSFUL_COMMIT to ensure the above script works correctly.

1. Navigate to the top level job where the git plugin is configured. Hint: inspecting a job at this level will include the GIT_* environment variables.
2. Configure the top level job.
3. Find the job build section and look for the job that you'd like to optionally skip.
4. Open advanced options for the job.
5. Add parameters: predefined parameters, and include the following at the very least.

GIT_COMMIT=${GIT_COMMIT}
GIT_PREVIOUS_SUCCESSFUL_COMMIT=${GIT_PREVIOUS_SUCCESSFUL_COMMIT}

Forcing a Build

Ok great, you got this far and the build task only runs when the file updates. But what if it doesn't run for a very, very long time? That won't be good, because if there is a problem then you'd prefer that there is time to handle any problems before deploying production. Sometimes third-party dependencies change and will introduce breaking changes to your build tasks. How about we force a build at a specific time? In this example, lets force it to build Mondays.

MONDAY=1
DAY_OF_WEEK=$(date +%u)

if [[ ${DAY_OF_WEEK} -eq ${MONDAY} || ! -z "$(sh jenkins/scripts/list_files_changed.sh | grep 'packer-ami.json')" ]]; then
    echo 'Run task: Its Monday or there are changes since last successful build.'
fi