Full CI integration for Git

Starting Configuration

Servers

Bitbucket: We shall use the URL http://BITBUCKET to refer to this server which shall have a project named http://BITBUCKET/projects/PROJECT_KEY/repos/REPOSITORY. This repo will have a folder containing the testing code named jenkins_scripts.
Jenkins: We shall use the URL http://JENKINS to refer to this server

Plugin installation

Not essential but nice to have

Bitbucket
- https://marketplace.atlassian.com/apps/1219256/commit-graph-for-bitbucket-server?hosting=server&tab=overview
Jenkins
- AnsiColor
- Generic Webhook Trigger
- Rebuilder

Set up Bitbucket on Jenkins

Under Credentials > System > Global credentials add a new set of credentials with the username and password of a bitbucket user with write access and the description Bitbucket credentials.

Under Manage Jenkins > Configure System > Bitbucket Endpoints remove the existing bitbucket instance and create a new one with the following parameters:

Name: Bitbucket
Server URL: http://BITBUCKET
Call Can Merge: Yes
Manage Hooks: Yes
Credentials: Bitbucket credentials

Conventions

Every release will be marked by a annotated git tag in the form release/TXX.XX.XX-XX or release/[SCV]XX.XX.XX. To build a new release, the following algorithm is followed:

Find the last commit with a tag starting with release/
Take the tag with the largest version number to be the last version
Take the numeric parts of the version, which will be in the format [TSCV]major.minor.bugfix(-prerelease)?. The prerelease is 0 if not present.
If you are not building a T release, set the prerelease to 0
If you are building a T increment the prerelease. If the last version was not a T release, increment the bugfix number.
If this version already exists, bail out

This scheme can be used to automatically deduce a releases version given only the git hash we want to build and the package letter we are building. The build screen prevents builds from having clashing version numbers, as the above algorithm only detects clashing version numbers if that version number is before the commit it is building.

To increase the major or minor version numbers, create a new tag a commit with:

git tag -a <version> <commit> -m "Build: None"
git push --tags

All builds on or after <commit> will increment from the new version number.

Every release branch will have the naming scheme release/XX.XX.

Run tests on commit

Creating the Jenkins job

Click on New Item and create a new Multibranch Pipeline (any name acceptable). Under:

Branch Sources add Bitbucket with the following values

Credentials: Bitbucket credentials
Owner: PROJECT_KEY
Repository Name: REPOSITORY
Behaviors: <leave at defaults and add Git > Advanced clone behaviors>

Branch Configuration set script path to jenkins_scripts/Jenkinsfile

Create a file named jenkins_scripts/Jenkinsfile with the contents:

pipeline {
    agent {
        label '<label>'
    }
    stages {
        stage('Testing') {
            steps {
                echo 'Starting test...'
                bat '''<test commands>
                '''
            }
        }
    }
}

where <test commands> is the commands to run your test suite, returning a non-zero exit code on failure. <label> is the Jenkins node label to run the tests on. If your tests can run on any making replace the agent section with agent any.

When you click on Scan Multibranch Pipeline Now jobs should start for the branches and pull requests your repo has that contain the jenkins_scripts/Jenkinsfile. This should also occur when you push a new commit. If it doesn't see the Hacks section at the end of this document.

Release Scripts

Create a new Jenkins Freestyle job with the name PROJECT_KEY-REPOSITORY-build. Tick This project is parameterised and add fields for GIT_HASH and PACKAGE_LETTER. Restrict by label is needed then under Source Code Management select Git with the following values:

Repository URL: http://BITBUCKET/scm/PROJECT_KEY/REPOSITORY.git
Credentails: Bitbucket credentials
Branches to build: origin/master

And add to the build section the necessary scripts. These scripts must do the following:

Update the tags in the git repo
Switch git to the commit GIT_HASH
Calculate based off the last tag the next version number
Build
Create a git annotated tag for that version (named with the above conventions) with the message Build: JOB_URL (it would be nice to display data from the job in the changelog at some point, however Bitbucket's API does not currently support querying the message of an annotated tag).

For an example script, see the LiveTouch Manager jenkins_scripts/Release.py. These steps will ensure compatibility with the changelog browser and builder.

Changelog Browser and Builder

This is a set of scripts which provide a familiar webpage detailing the changelog for each version and allowing builds to be started. It is located at \\cimaster\htdocs\gitchangelog. To view the changelog for a particular repository go to http://cimaster/gitchangelog/PROJECT_KEY/REPOSITORY. If the project follows the conventions set out in this document, you will be able to build releases from that page as well as see the changelog in the same format at the previous Clearcase system.

The system requires a Bitbucket personal access token with the read privileges to be places in the var BITBUCKET_ACCESS_TOKEN = "..."; line.

Locations that require updates

When the Bitbucket URL changes:

Jenkins > Configure System: Detailed under Set up Bitbucket on Jenkins
Jenkins > Each release job: Change URL used in the Repository URL field in the Source Code Management section
Changelog Browser and Builder: Change URL in \\cimaster\htdocs\gitchangelog\git.js on the var BITBUCKET_URL = "..."; line

When the Jenkins URL changes:

Bitbucket > Each repo > Settings > Post Webhooks: If you set any webhooks manually, update them
Changelog Browser and Builder: Change URL in \\cimaster\htdocs\gitchangelog\git.js on the var JENKINS_URL = "..."; line

When the Bitbucket authentication changes:

Changelog Browser and Builder: Generate a new read access token and place it in \\cimaster\htdocs\gitchangelog\git.js on the var BITBUCKET_ACCESS_TOKEN = "..."; line
Update the credentials in Jenkins > Credentials > System > Global credentials > Bitbucket credentials

When the Jenkins authentication changes:

Go to the accounts settings, and reveal the API token, and place it with the username at \\cimaster\htdocs\gitchangelog\git.js on the var JENKINS_ACCESS_CREDENTIALS = "..."; line

Hacks

Jobs are not run on push

The webhook that Jenkins sets up may not point to the correct server if Jenkins is misconfigured. On the repo go to Settings > Post Webhooks (not Settings > Webhooks) and add a new webhook with the following values:

Title: Jenkins
URL: http://JENKINS/bitbucket-scmsource-hook/notify

And tick all the checkboxes except: Tag Created, Build Status and Pull request commented.

matsjoyce/cibuild.md