CircleCi - only build features that has changed

buildWhenAffected.sh

##################
### config.yml ###
##################

version: 2
jobs:
    build:
        docker:
            - image: circleci/python:3.6
        steps:
            - checkout
            - run:
                command: |
                    .circleci/commit_check.sh



#######################
### commit_check.sh ###
#######################

set -e

# latest commit
LATEST_COMMIT=$(git rev-parse HEAD)

# latest commit where path/to/folder1 was changed
FOLDER1_COMMIT=$(git log -1 --format=format:%H --full-diff path/to/folder1)

# latest commit where path/to/folder2 was changed
FOLDER2_COMMIT=$(git log -1 --format=format:%H --full-diff path/to/folder2)

if [ $FOLDER1_COMMIT = $LATEST_COMMIT ];
    then
        echo "files in folder1 has changed"
        .circleci/do_something.sh
elif [ $FOLDER2_COMMIT = $LATEST_COMMIT ];
    then
        echo "files in folder2 has changed"
        .circleci/do_something_else.sh
else
     echo "no folders of relevance has changed"
     exit 0;
fi

How would this work for a project with several independent microservices?
If the project root had 4 projects in it: ./web, ./api, ./worker, ./admin and we only wanted to build the microservices that changed.

Would it be better to use workflows, or would the script above work for this?

This is exactly the use case this was made for. Sorry for the late response, @KingScooty. Without a mention I dont get notified.

@naesheim . This doesnt fully work. I've got the exact scenario as KingScooty and the commits are different even through they're not

@naesheim How can we integerate above script with CircleCI workflow?

@michellaporte, are all the target folders included in the git repository?

Ideally, 'git log -1 --format=format:%H --full-diff my/special/path' should show the commit SHA when my/special/path was last changed. Is this not the case?

@rahulwa
In my config.yaml I only have one stage, which is 'build'. If I also had a stage called 'test'. It would have been something like this:

workflows:
    version: 2
    build_and_test:
       jobs:
         - build
         - test

how to you handle merge commits?

In our workflow we only build on master, and every developer send a pull requests, and then we have a merge commit to get the changes on master

what about using CIRCLE_COMPARE_URL?

check this https://github.com/entria/entria-deploy

how to you handle merge commits?

In our workflow we only build on master, and every developer send a pull requests, and then we have a merge commit to get the changes on master

yes, git log -1 --format=format:%H --full-diff path/to/folder1 can be problematic when dealing with merge commits https://haacked.com/archive/2014/02/21/reviewing-merge-commits/

what about using CIRCLE_COMPARE_URL?

Thanks! I did not know about this one:)

check this https://github.com/entria/entria-deploy

I know there are different ways of handling this. There seem to pop up new tools to handle builds on monorepos every day. I'm working on a blog post about it.

Here's another approach that I've taken, for other people's reference:

Write a script with the following steps:

• get the last run's commit from CIRCLE_COMPARE_URL
• get the current run's commit from CIRCLE_SHA1
• run git diff <last_run_commit> <current_run_commit> <dir> 2>&1
• if the above result is an empty string, the script prints False, otherwise it prints True

---

I'm using python executor so the script is in python.
You can choose to use whatever language you like, shell script is OK too.

Example usage of it would be:

CHANGE_CHECKER=./scripts/changed-since-last-commit.py
SUB_DIR=./src/api
if [[ `python3 ${CHANGE_CHECKER} ${SUB_DIR}` == "True" ]]; then
    # run unit tests against ${SUB_DIR}
fi

This script above is designed to be run every time, because the logic whether to run the real task or not is inside the script.

---

changed-since-last-commit.py:

import os
import sys


def git_compare_url() -> str:
    # something like: https://github.com/mkobit/gradle-test-kotlin-extensions/compare/211a8ef37eb6^...3c546b55628a
    return os.getenv('CIRCLE_COMPARE_URL', '')


# https://discuss.gradle.org/t/build-scan-plugin-1-10-3-issue-when-using-a-url-with-a-caret/24965
def get_last_run_commit() -> str:
    return git_compare_url().split('/compare/')[1].split('...')[0].replace('^', '')


def get_current_run_commit() -> str:
    return os.getenv('CIRCLE_SHA1', '')


def changed_since_last_run_commit(dirs: list) -> bool:
    if not git_compare_url():
        # if we cannot parse the previous revision, we believe current commit has changes in the directory
        return True
    else:
        # We redirect stderr to stdout. If there is an error (e.g. in the first pipeline or some other edge cases), we regard everything as changed
        cmd = 'git diff "%s" "%s" %s 2>&1' % (get_last_run_commit(), get_current_run_commit(), ' '.join(dirs))
        result = os.popen(cmd).read()
        return not not result


if __name__ == "__main__":
    changed = changed_since_last_commit(sys.argv[1:])
    print(changed)

Hope it helps ~

@sinogermany awesome stuff!

Looks like CircleCI used with pipelines disables CIRCLE_COMPARE_URL variable. This might get handy then https://github.com/iynere/compare-url

@ondrejsevcik With pipelines there are those now, which are even better: https://circleci.com/docs/2.0/pipeline-variables/#pipeline-values. Thank you everyone for the ideas discussed here.

The original solution does not work correctly if you have multiple commits in a branch. The scenario is: edit a file in the path you are checking, commit it, then make a change to another path, commit it, then push both commits to Circleci (if you have it set up to only run for the last commit in a branch). It will check the last commit which does not have the changes to the path you're checking.

In order to know if something changed when there are multiple commits, you need to have a base commit/sha to compare to (called merge base). You could assume all changes eventually get to origin/main (this may not true for all cases, like a shared topic/feature branches), and use something like this bash script:

#!/bin/bash

REF=HEAD
SINCE=origin/main
DIR_TO_CHECK=path/to/your-directory

MERGE_BASE=$(git merge-base ${SINCE} ${REF})
FILES_CHANGED=$(git diff --name-only ${MERGE_BASE}..${REF} -- ${DIR_TO_CHECK})
printf "Files changed:\n${FILES_CHANGED}\n"

if [[ -n $FILES_CHANGED ]]; then
  echo "Found changes"
else
  echo "No changes"
fi

Circle CI has since came up with their own recommended way to do this:
https://circleci.com/docs/2.0/configuration-cookbook/?section=examples-and-guides#execute-specific-workflows-or-steps-based-on-which-files-are-modified

The circlci example at https://circleci.com/docs/2.0/configuration-cookbook/?section=examples-and-guides#execute-specific-workflows-or-steps-based-on-which-files-are-modified returns below error:

[#/jobs/check-updated-files] 0 subschemas matched instead of one
1. [#/jobs/check-updated-files] only 0 subschema matches out of 2
|   1. [#/jobs/check-updated-files] no subschema matched out of the total 2 subschemas
|   |   1. [#/jobs/check-updated-files] 0 subschemas matched instead of one
|   |   |   1. [#/jobs/check-updated-files] expected type: Mapping, found: Sequence
|   |   |   |   SCHEMA:
|   |   |   |     type: object
|   |   |   |   INPUT:
|   |   |   |     - filter:
|   |   |   |         mapping: |
|   |   |   |           service1/.* run-build-service-1-job true
|   |   |   |           service2/.* run-build-service-2-job true
|   |   |   |         base-revision: master
|   |   |   |         config-path: .circleci/config.yml

i get the same ^