EE4J Optional Pipelines

We do want 2 kind of CI pipelines:

Gate keeper that runs for everything, protected branches, feature branches and pull requests
Optional pipeline run on a "on demand" basis, controlled by committers and based on the changesets content

For GlassFish, the gate keeper pipeline shall run as fast as possible with the best coverage possible. Test suites are split in small chunks and executed concurrently, this allows to run many test suites and keep the end-to-end pipeline time acceptable.

Theoretically we could run every single test suite for every changeset ; however there is an implicit limitation of compute resources where the build could slow down the end-to-end time. Test suites may also not be 100% stable, causing sporadic failures that require a manual retry of some sort.

We need the ability to run additional test suites on a "on demand" basis, e.g. CTS. This shall be driven by review comments on the PR. Only comments from members of whitelisted GitHub organization should trigger new pipeline runs.

In our case we'd whitelist the "eclipse-ee4j" GitHub organization, this would allow non GlassFish committer to trigger optional pipelines and maybe also retry failed pipelines. This should make the overall process more self-manageable. Note that currently only project committers can trigger builds ; using this comment based approach would solve this nicely.

Pipeline as Code

Every pipeline / job should be checked-in under their respective project source repository, optional or not. Test suites should be integrated like anything else ; with a fixed version that makes the process reproducible.

Triggering pipelines across Jenkins instance breaks this model. E.g. If a project triggers a multi-branch pipeline from another repository, this means it actually references code (the pipeline) from a branch in that repository ; any bad commit pushed to that branch can potentially break the CI for the project without a single code change. This is bad and SHOULD BE AVOIDED.

CTS Pipelines

Each project that need to run CTS shall write their own pipeline to run the CTS. Instructions should be provided by the CTS/GlassFish team to help with that.

The CTS team will provide a Docker image that can be used to run any CTS test suite
The Docker image will contain script(s) to bootstrap and run any CTS test suite
Each project CTS pipeline will be setup to run relevant CTS suites

Implementation notes

Disable Push Triggers for Optional Pipelines

Multi-branch pipelines are by default triggered for push events and pull_request events. This is not desireable for optional pipelines since we want to trigger them based-off comments.

This can be done as follow:

Go the "Configure" screen for the top level multi-branch pipeline project
In the "Branch Source" section, click "Add Property" then add "Suppress Automatic SCM triggering"

The result will be that the pipeline won't be triggered by any GitHub Webhook event or repository scanning.

Implement a Trigger Job

This job requires the "Github Pull Request Builder Plugin" and the "Parameterized Trigger Plugin" ; both plugins are installed in the Eclipse Jenkins for GlassFish.

Create a pipeline job, then configure it as follow:

In the "General" section, configure the "GitHub Project": fill-in the project URL
In the "Build Triggers" section, check the "GitHub Pull Request Builder"

Add a trigger phrase ; e.g. "^/(re)?run .*"
Check "Only use trigger phrase for build triggering"
Set "Skip build phrase" to empty
Add two trigger setups
- Build status Messages: add one entry for every status (success, error, failure) and put "--none--" as message
- Update commit status during build: set every field to "--none--"

This approach works but is relatively static: one type of comment can only trigger one set of pipelines. If we want to be more sophisticated we may want to investigate a scripted pipeline with groovy logic to handle triggers dynamically based on comments.

Add the following classes to JENKINS_URL/scriptApproval/:

method hudson.model.Run getCauses
method org.jenkinsci.plugins.ghprb.GhprbCause getCommentBody
method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild

Setup the pipeline section as follow:

Set "Definition" to "Pipeline script from SCM"
Configure the Git repository as follow

Add the HTTPS URL of the Git repository
Click "Advanced, set "Refspec" to "+refs/pull/:refs/pull/"
Set "Branches to build" to "refs/pull/${ghprbPullId}/head"

Set "Script Path" to "ProcessTrigger.groovy"

Create a script in the repository named ProcessTrigger.groovy:

node {
    stage('process-trigger') {
        def commentBody = get_comment_trigger();
        if(commentBody != null){
            // this is illustrative.
            // the ghprb plugin is configured with a loose regexp
            // trigger different job based on the comment content
            if(commentBody.contains("tests")){
                build job: "cts-pipeline/PR-${ghprbPullId}"
            } else {
                // fail here if the comment content is supported
                error("Unsupported comment message")
            }
        } else {
            // only process comment based trigger
            error("Unsupported trigger")
        }
    }
}

@NonCPS
def get_comment_trigger() {
    def causes = currentBuild.rawBuild.getCauses()
    if(causes.size() > 0
        && causes[0].getClass().getName() == "org.jenkinsci.plugins.ghprb.GhprbCause"){
        return currentBuild.rawBuild.getCauses()[0].getCommentBody()
    }
    return null;
}

Pipeline re-build

Leverage the dynamic processing of trigger comments to implement re-triggering the "GateKeeper" pipeline.

romain-grecourt/ee4j-optional-pipelines.md