Understand the Basics of Reusable Workflows in GitHub Actions

gist-reusable-gh-workflows.md

The Basics of GitHub Actions Reusable Workflows

---

Duplication and repetition is pretty common in infrastructure configuration languages like GitHub Actions. Although you can never seem to get rid of all of it, there is usually something you can do to get rid of at least some of it.

If you use GitHub Actions in your GitHub repositories, like for your Continuous Integration/Continuous Deployment (CI/CD), you may find that you have a lot of repetition and duplication in your Actions workflows.

This is especially true if you have several repositories with the same workflows.

To reduce some of this duplication and the costs and hassles that come with it, you can use Reusable Workflows.

Because of the limitations listed below and demonstrated later, reusable workflows will not eliminate all of your duplication, but it definitely helps and provides solid value in easier maintenance.

In this post I will show you the basics of Reusable Workflows in GitHub Actions, enough to get you going with your own reusable workflows. Their Limitations. How they work. How they are structured. How to call and use them. How to get information in and out of them. Finally, (and you are totes free to stop reading by this point) my perspective on testing and design and development with them.

🙇 GitHub Documentation on Reusing Workflows

---

Limitations

It is always good to start with what you can't do. Get it right out in the open right away. Besides you will revisit most of this again in specifics later in this post.

The limitations of reusable workflows...

• Reusable workflows can not call reusable workflows, you can not nest reusable workflows

• Reusable workflows in private repositories can only be used by workflows in the same repository

• Caller workflow environment variables are not available to the called workflow

  • You can not set reusable workflow inputs from caller environment variables
  • Any environment variables set in an env context defined at the workflow level in the caller workflow are not propagated to the called workflow.

• Any job that calls a reusable workflow can not use strategy property i.e. build matrix

🙇 GitHub Documentation on Limitations of Reuseable Workflows

---

Background

This covers some of the background and an overview of reuseable workflows...

• What does a workflow have to have to be reuseable
• The two sides of a reusable workflow, the caller and called
• What happens when a reusable workflow runs

What Workflows Can Be Reused

What makes a workflow able to be reused?

1. Reusable workflow files are basically the same as other GitHub Actions YAML workflow files and must be located in the relative directory (no subdirectories)...

   .github/workflows/
   

2. You can call a reusable workflow in your workflow IF either...

   • The caller and called workflows are in the same repository
   • The called workflow is in a public repository

🙇 GitHub Documentation on Access to Reusable Workflows and GitHub Documentation on Calling a Reusable Workflow

The Two Sides of Reusable Workflows

There are two sides or perspectives in reusing workflows...

1. The caller workflow is the workflow that is using the reusable workflow

2. The called workflow is the reusable workflow being, well, called

Understanding these two perspectives is critical to knowing what will be available when running your reusable workflows.

What Happens When You Run a Reusable Workflow

When running a reusable workflow in a calling workflow...

• The entire called workflow (all jobs) is used

• The called workflow runs like it was part of the caller workflow e.g. called actions/checkout checks out the contents of the caller repository hosting the caller workflow

• The called workflow Github Context is the caller's GitHub Context

• The called workflow is automatically granted access to github.token and secrets.GITHUB_TOKEN of the caller workflow

🙇 GitHub Documentation Reusable Actions Overview

---

An Example of a Reusable Workflow

This post contains examples from my reusable CI/CD workflows which are called and used in my personal projects.

Examples of Reusable Workflows

For examples of reusable workflows, reference the workflows under https://github.com/brianjbayer/actions-image-cicd/tree/main/.github/workflows

Note that the workflows that begin with on_ are calling workflows that call and exercise the reusable workflows in that project.

Examples of Workflows That Use Reusable Workflows

For examples of calling workflows that use my reusable CI/CD workflows, you can reference the workflows that begin with on_ in https://github.com/brianjbayer/actions-image-cicd/tree/main/.github/workflows or see the workflows at https://github.com/brianjbayer/sample-login-capybara-rspec/tree/main/.github/workflows.

---

The Structure of a Reusable Workflow

Here is an example of a reusable (or called) workflow named...

.github/workflows/git_github_info.yml

This example happens to show that you can use other actions and have multiple steps in your job.

This example only has a single job, but reusable workflows can have multiple jobs which will all run.

name: Display Basic Git and GitHub (Actions) Info

on:
  workflow_call:

jobs:
  git-github-info:
    name: Git and GitHub Information
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v1

      - name: Git log
        run: git log

      - name: Show GitHub context
        env:
          GITHUB_CONTEXT: ${{ toJson(github) }}
        run: echo "$GITHUB_CONTEXT"

This example shows that the structure of a reusable workflow file is mostly that of a basic GitHub Actions workflow file with a few nuances.

😁 You can refresh yourself on the basics of GitHub Actions workflows with my post The Basics of GitHub Actions Workflows

Triggering a Reusable Workflow With on: workflow_call:

One of the main differences of a reusable workflow is that it is not triggered by an event, but by being called. This is why the on (the trigger configuration) is different.

To trigger the reusable workflow by calling it, set the on: to workflow_call:.

on:
  workflow_call:

With reusable workflows, it is the calling workflow that is triggered by events such as a Pull Request (PR) which will then call your reusable workflow.

Inside a Running Reusable Workflow

Although simple and self-contained with no inputs or outputs, this example reusable workflow does illustrate the two perspectives of a called reusable workflow.

Consider the single job in the reusable workflow...

    steps:
      - uses: actions/checkout@v1

      - name: Git log
        run: git log

      - name: Show GitHub context
        env:
          GITHUB_CONTEXT: ${{ toJson(github) }}
        run: echo "$GITHUB_CONTEXT"

In the first step, it uses: actions/checkout@v1, showing that you can use other "Actions".

    steps:
      - uses: actions/checkout@v1

But another important point is that when this action runs, it will "run like it was part of the caller" so it will do a git checkout of the caller's git repository. This then allows the next step to output the git log for the caller's repository.

🚢 Captain Obvious says that this demonstrates how when the reusable workflow runs, it has the caller's perspective.

A final important point that this example reusable workflow shows...

      - name: Show GitHub context
        env:
          GITHUB_CONTEXT: ${{ toJson(github) }}
        run: echo "$GITHUB_CONTEXT"

When this step in the reusable workflow runs, it will display the GitHub Context which will be the caller's Context. This means that your reusable workflows have access to the caller's GitHub Context (e.g. GITHUB_CONTEXT: ${{ toJson(github) }}). This also is how you can display the values and fields in your GitHub Context in printable form.

🚢 Captain Obvious says that it is having the caller's perspective that makes your reusable actions more reusable.

BUT one thing that is not available to the reusable workflow is any of the caller's environment variables as mentioned in Limitations.

---

Using a Reusable Workflow

Now here is an example of a workflow file (maybe in another repository) that calls the reusable workflow...

name: On PR Workflow Checks

on:
  pull_request:
    branches:
      - main

jobs:

  check-git_github_info:
    uses: brianjbayer/actions-image-cicd/.github/workflows/git_github_info.yml@main

This shows that the calling workflow has the GitHub event trigger (in this case, on PRs against the main branch).

Calling the Reusable Workflow With uses

It then defines it's own job check-git_github_info that calls or uses the reusable workflow specifying the reference.

jobs:

  check-git_github_info:
    uses: brianjbayer/actions-image-cicd/.github/workflows/git_github_info.yml@main

There are a couple of things that you should know about calling a reusable workflow in another (calling) workflow...

• You must specify an @{ref} (e.g. @main) when using a reusable workflow

• The only thing that the calling job can do is call (i.e. use ) a reusable workflow, it can not have steps and is basically a "proxy" job for the reusable workflow in the calling workflow

👀 About that @{ref}: You must specify an @{ref} when useing a reusable workflow. It can be a SHA, a release tag, or a branch name. The SHA is the safest, most stable, and clearly defined.

You can refer to the reusable workflow file using either of the following two manners...

1. For reusable workflows in a public repository...

   {owner}/{repo}/{path}/{filename}@{ref}
   

2. For reusable workflows in the same repository...

   ./{path}/{filename}
   

   Note that here the reusable workflow is from the same commit as the caller workflow

🙇 GitHub Documentation on Calling Reusable Workflows

Calling Dependent Reusable Workflows and Other Capabilities

Calling a reusable workflow can be dependent on another job in that caller workflow just like in basic workflows with needs.

  check-pull_push_image:
    needs: [check-build_push_image]
    uses: brianjbayer/actions-image-cicd/.github/workflows/pull_push_image.yml@main

In this example, the calling workflow job check-pull_push_image (which calls the reusable workflow .github/workflows/pull_push_image.yml) is dependent on (needs) the successful completion of calling workflow job check-build_push_image.

✨ For all of the capabilities available when calling reusable workflows, see the GitHub Documentation on Supported keywords for jobs that call a reusable workflow

---

Getting Information Into Reusable Workflows (inputs and secrets)

To be truly flexible and reuseable, you will need a way of getting information into your reuseable workflows. Information like input values and secrets to access the caller's secure systems like image repositories.

For this reason, reusable workflows have inputs and secrets

🙇 GitHub Documentation on inputs and secrets in a reusable workflow

Inputs

inputs are how you pass input values to your reusable workflows from the calling workflow.

There are three parts to passing inputs to a reusable workflow...

1. In the reusable workflow, define the inputs
2. In the reusable workflow, reference and use the inputs
3. In the calling workflow, pass the inputs to the reusable workflow

Getting Inputs Into the Reusable Workflow With inputs

In the reusable workflow file, the inputs are first defined and then used. Here is an example from part of a reusable workflow .github/workflows/build_push_image.yml that uses inputs.

First, define the inputs in the reusable workflow using inputs.

name: Idempotent Build Push Image

on:
  workflow_call:
    inputs:
      image:
        required: true
        type: string
      buildopts:
        type: string

Here we have defined two inputs (variables) image and buildopts both strings. The required: true specifies that the image input is not optional and must be supplied when calling the reusable workflow.

Second, reference and use the defined inputs in the reusable workflow using the format ${{ inputs.<inputs-name> }} e.g. ${{ inputs.image }}.

jobs:

  pull-or-build-and-push-image:
    name: Pull or Build and Push Image
    runs-on: ubuntu-latest
    env:
      IMAGE: ${{ inputs.image }}

    steps:
      - uses: actions/checkout@v1

      - name: Image name
        run: echo "Image name [${IMAGE}]"

Here we happen to define a job-level environment variable IMAGE with the value of the input ${{ inputs.image }} and then later output it.

✨ This does show that you can set environment variables from inputs in reusable workflows

Passing Inputs With with When Calling the Reusable Workflow

The final part of inputs with reusable workflows is setting them when they are called in another workflow. To include inputs when you call a reusable workflow, use with.

Here is our reusable workflow with inputs .github/workflows/build_push_image.yml being called in another workflow...

name: On PR Workflow Checks

on:
  pull_request:
    branches:
      - main

env:
  BRANCH: ${{ github.head_ref }}
  COMMIT: ${{ github.event.pull_request.head.sha }}

jobs:

  check-build_push_image:
    uses: brianjbayer/actions-image-cicd/.github/workflows/build_push_image.yml@main
    with:
      image: ${{ github.repository }}_${{ github.head_ref }}_test:${{ github.event.pull_request.head.sha }}

Here the image input is set with values from the GitHub Context. Note that the value is explicitly set with the GitHub Context values and not with the calling workflow's environment variables.

👀 Here is an example of not being able to use environment variables in the calling workflow with reusable workflows. Although the BRANCH and COMMIT variables are set, they can not be used to set the reusable workflow inputs (they appear as unset/empty in the reusable workflow)

Secrets

Often your reusable workflows need secure access to the calling workflow's secrets (for example a reusable workflow that builds an image from the calling repository and pushes it to an image registery as part of CI).

Using secrets with a reusable workflow is basically the same as inputs with the same three parts...

1. In the reusable workflow, define the secrets
2. In the reusable workflow, reference and use the secrets
3. In the calling workflow, pass the secrets to the reusable workflow

Getting Secrets Into the Reusable Workflow With secrets

In the reusable workflow file, the secrets are first defined and then used.

Here is an example from part of the reusable workflow .github/workflows/build_push_image.yml that uses secrets.

name: Idempotent Build Push Image

on:
  workflow_call:
    inputs:
      image:
        required: true
        type: string
      buildopts:
        type: string
    secrets:
      registry_u:
        required: true
      registry_p:
        required: true

jobs:

  pull-or-build-and-push-image:
    name: Pull or Build and Push Image
    runs-on: ubuntu-latest
    env:
      IMAGE: ${{ inputs.image }}

    steps:
      - uses: actions/checkout@v1

      - name: Image name
        run: echo "Image name [${IMAGE}]"

      - name: Login to DockerHub Registry
        run: echo ${{ secrets.registry_p }} | docker login -u ${{ secrets.registry_u }} --password-stdin

Along with the inputs from before, this reusable workflow defines the required secrets registry_u registry_p.

on:
  workflow_call:
    inputs:
      image:
        required: true
        type: string
      buildopts:
        type: string
    secrets:
      registry_u:
        required: true
      registry_p:
        required: true

It then uses them in the step to Login to the DockerHub image registry.

      - name: Login to DockerHub Registry
        run: echo ${{ secrets.registry_p }} | docker login -u ${{ secrets.registry_u }} --password-stdin

Passing Secrets With with When Calling the Reusable Workflow

The final step in using secrets with your reusable workflows is passing the calling workflow's secrets to the reusable workflow. Like inputs, use with to include setting secrets when calling a reusable workflow.

Here is the example workflow being called with its secrets being set by the calling workflow.

  check-build_push_image:
    uses: brianjbayer/actions-image-cicd/.github/workflows/build_push_image.yml@main
    with:
      image: ${{ github.repository }}_${{ github.head_ref }}_test:${{ github.event.pull_request.head.sha }}
    secrets:
      registry_u: ${{ secrets.DOCKER_HUB_USERNAME }}
      registry_p: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}

Here the reusable workflow secrets registry_u and registry_p are being set to the calling workflow's secrets ${{ secrets.DOCKER_HUB_USERNAME }} and ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }} respectively.

---

Getting Information Out of Reusable Workflows With outputs

🙇 GitHub Documentation on Using Outputs

When you need information from a reusable workflow, probably to use in other workflows, you use outputs.

Although overall similar to inputs and secrets, outputs are a bit different (and a little more work).

At a high level, outputs still have the same three parts...

1. In the reusable workflow, define the outputs
2. In the reusable workflow, set the values of the outputs
3. In the calling workflow, get the values of the outputs

But the key difference with outputs is that the output is set at the step level in a job but passed at the reusable workflow level.

In the Reusable Workflow

Since the reusable workflow output is set at the step level in a job but passed at the workflow level, there are a couple of places in the reusable workflow where you define and set outputs (some intermediate).

You will first define the output at the workflow level and set it from the output of a job in the reusable workflow.

Then you will define the job-level output and set it from an identified step output.

Finally you will set the step-level output in the identified job.

Consider this example of reusable workflow file .github/workflows/get_merged_branch_last_commit.yml

name: Merged Branch and Last Commit SHA of Branch

on:
  workflow_call:

    outputs:
      branch:
        value: ${{ jobs.merged-branch-commit.outputs.branch }}
      commit:
        value: ${{ jobs.merged-branch-commit.outputs.commit }}

jobs:
  merged-branch-commit:
    name: Generate merged branch and branch last commit
    runs-on: ubuntu-latest
    outputs:
      branch: ${{ steps.getbranch.outputs.branch }}
      commit: ${{ steps.getcommit.outputs.commit }}

    steps:
      - uses: actions/checkout@v1

      - id: getbranch
        run: echo "branch=$(git log -1 --pretty=%B | grep 'Merge pull request' | sed 's/^[^/]*\///')" >> $GITHUB_OUTPUT
      - id: getcommit
        run: echo "commit=$(git log -n 1 --skip 1 --pretty=format:"%H")" >> $GITHUB_OUTPUT

Here there are are two workflow outputs defined,branch and commit and their values are set from the outputs branch and commit of a job named merged-branch-commit (i.e. branch: ${{ steps.getbranch.outputs.branch }})

name: Merged Branch and Last Commit SHA of Branch

on:
  workflow_call:

    outputs:
      branch:
        value: ${{ jobs.merged-branch-commit.outputs.branch }}
      commit:
        value: ${{ jobs.merged-branch-commit.outputs.commit }}

Then later in the workflow, the two job outputs branch and commit for the merged-branch-commit job are defined and set from the output of steps in the job with the ids of getbranch and getcommit.

jobs:
  merged-branch-commit:
    name: Generate merged branch and branch last commit
    runs-on: ubuntu-latest
    outputs:
      branch: ${{ steps.getbranch.outputs.branch }}
      commit: ${{ steps.getcommit.outputs.commit }}

Finally in the steps of the jobs, the output values are actually set with the syntax run: echo "{name}={value}" >> $GITHUB_OUTPUT.

      - id: getbranch
        run: echo "branch=$(git log -1 --pretty=%B | grep 'Merge pull request' | sed 's/^[^/]*\///')" >> $GITHUB_OUTPUT
      - id: getcommit
        run: echo "commit=$(git log -n 1 --skip 1 --pretty=format:"%H")" >> $GITHUB_OUTPUT

Here we are executing a pipeline of unix commands to set the values of the outputs.

Originally GitHub Actions used :set-output as the output method, however this was deprecated for setting the GITHUB_OUTPUT environemt variable

In the Calling Workflow

In the calling workflow, there are at least two jobs associated with the outputs of the reusable workflow.

The first job calls the reusable workflow and serves as the holder or "proxy" for the outputs.

The second or more jobs actually use the outputs from the first job.

Here is an example from a workflow that calls our reusable workflow with outputs.

  branch-and-last-commit:
    uses: brianjbayer/actions-image-cicd/.github/workflows/get_merged_branch_last_commit.yml@main

  branch-and-last-commit-merged-info:
    needs: [branch-and-last-commit]
    runs-on: ubuntu-latest
    env:
      BRANCH_LAST_COMMIT: ${{ needs.branch-and-last-commit.outputs.commit }}
      BRANCH: ${{ needs.branch-and-last-commit.outputs.branch }}

Here is the first job that calls the reusable workflow with outputs...

  branch-and-last-commit:
    uses: brianjbayer/actions-image-cicd/.github/workflows/get_merged_branch_last_commit.yml@main

Then the second job is dependent on (needs) the first job and uses its outputs...

  branch-and-last-commit-merged-info:
    needs: [branch-and-last-commit]
    runs-on: ubuntu-latest
    env:
      BRANCH_LAST_COMMIT: ${{ needs.branch-and-last-commit.outputs.commit }}
      BRANCH: ${{ needs.branch-and-last-commit.outputs.branch }}

Here the outputs are being used to set the value of the second job's environment variables.

This concludes the basics of GitHub Actions workflows and you should now be able to start developing and using your own reusable workflows or at least better understand them.

---

My Thoughts on Testing and Developing Reusable Workflows

In the remainder of this post, I will present my initial thoughts on testing and developing based on my experience with reuseable workflows.

Testing Reusable Workflows

As I mention in my post on the The Basics of GitHub Actions Workflows there is no native testing facility for GitHub Actions and this is also the case for reusable workflows. This makes testing your workflows rather painful.

Limiting your testing even more is that you can not really call scripts in your reusable workflows as these scripts would need to be in the repository of every calling workflow that uses the reusable workflow.

Thus the only two practical ways to test your reusable workflows as well as the calling workflows are..

1. Putting the reusable workflow under test in an on-Pull-Request calling workflow letting you run it when pushing a new commit on a Pull Request (PR). However, this approach will not help if there is any data or state that is only present during merges or other events.

2. Creating a separate (and disposable) GitHub repository as a testbed for your calling and reusable workflows. This is especially useful when developing and testing merge workflows.

---

Designing and Developing Reusable Workflows

There is definitely value in reusable workflows for all of their limitations, but those limitations really do hurt you in the amount of duplication and repetition ("copy pasta" and "cargo culting") and cost that you can eliminate in your GitHub Actions workflows, especially if you have basically the same workflows in multiple GitHub repositories.

About Those Limitations

These are the specific limitations that I encountered.

Reusable workflows can not call reusable workflows

This limitation really kills the levels of duplication that you can remove and your design options.

Consider that I have several GitHub Repositories that have basically the same CI/CD and the same workflows. These workflows have the same triggers, environment variables, jobs, job dependencies, and steps.

There is repetition at the repository levels at the overall CI/CD logic of multiple workflows. Within a workflow, there is repetition in the jobs and their similar but a little different steps.

But since reusable workflows can not call reusable workflows, you can only tackle one of these layers of duplication.

You can design your reusable workflows at the high-level repository business-logic level (e.g. CI/CD) but then still have duplication in your jobs, or you can design your reusable workflows at the lower utility job level and have duplication at the calling repository level (especially across repositories).

I generally choose to implement at the lower utility level with the hope of more reuse and greater cohesion.

Caller workflow environment variables are not available to the called workflow

This also hits you in the amount of code duplication and cost you can reduce. This one really hurts if you have a lot of the same inputs and values in your calling workflow especially if they are composites of other values like from GitHub Contexts or strings. For me, it is my branch-based, commit-tagged image names.

Other Considerations

There is also the balancing of cohesion and making the smallest unit of functionality and scalability versus the operational effectiveness and efficiency of doing multiple things in multiple steps.

Since each job runs on its own worker, that is the level of parallelism (and separation like environment variables). But this level of parallelism is limited to the number of Runners available (and this is where scalability comes into play by adding more runners).

Every time a job starts, there is a cost in the time that it takes to start up the machine and job. There is also a cost for that runner. So from a financial perspective, it makes sense to be efficient in "unit costs" and run as many jobs and as many steps as possible on a single runner.

This then is the struggle. I personally lean towards the side of cohesion and small utility workflows, but I always try to do what makes sense for the situation. In the outputs example, .github/workflows/get_merged_branch_last_commit.yml gets both the branch name and the commit instead of 2 separate jobs. In my use case for this reusable workflow and information, I always needed both of these things so it did make the decision a little easier.

---

This concludes this post on reusable workflows, I hope that it may help you or at least that you found something useful in it.

---

This is great, thank you for writing it up. I wanted to point out that it looks like nested reusable workflows are supported now:

https://github.blog/changelog/2022-08-22-github-actions-improvements-to-reusable-workflows-2/

https://docs.github.com/en/actions/using-workflows/reusing-workflows#nesting-reusable-workflows

@russell-mcdaniel

Hey Russell, thank you for the kind words and letting me know that nested reusable workflows are now supported. This is great and I really appreciate you sharing those links (especially the changelog). I will try and get this blog updated with this new information (and credit you of course 😄 ). I also have some refactoring to do in some of my projects 😉.