Tutorial and tips for GitHub Actions workflows
GitHub Actions is a CI/CD service that runs on GitHub repos.
Compared with Travis CI, GitHub Actions is:
- Easier
- More flexible
- More powerful
- More secure
- Workflows are YAML files stored in the .github/workflows directory of a repository.
- An Action is a package you can import and use in your workflow. GitHub provides an Actions Marketplace to find actions to use in workflows.
- A job is a virtual machine that runs a series of steps. Jobs are parallelized by default, but steps are sequential by default.
- To get started:
- Navigate to one of your repos
- Click the "Actions" tab.
- Select "New workflow"
- Choose one of the starter workflows. These templates come from actions/starter-workflows.
- Workflows can be triggered by many different events from the GitHub API.
- GitHub provides a context and expression syntax for programmatic control of workflows. For example:
echo ::set-output name=SPAM_STRING::${{ format( 'Spam is short for {0} and is made from {1} by {2}', 'spiced ham', 'pork shoulder', 'Hormel' ) }}
- Command:
echo ::set-output name=ENV_NAME::value
, likeecho ::set-output name=COLOR::green
- Expression:
${{ }}
- Function:
contains('this is a demo', 'demo')
evaluates to Booleantrue
format('Spam is short for {0} and is made from {1} by {2}', 'spiced ham', 'pork shoulder', 'Hormel')
- Command:
A workflow file might look like this:
name: demo
on:
push:
branches: [demo]
pull_request:
workflow_dispatch:
env:
APP_NAME: "GitHub Actions demo workflow"
jobs:
simple:
runs-on: ubuntu-latest
steps:
- name: Checkout repo
uses: actions/checkout@v2
- name: Verify the workspace context
run: echo 'Workspace directory is ${{ github.workspace }}'
- name: Run a simple echo command with a pre-set environment variable
run: echo 'Hello World, from ${{ env.APP_NAME }}'
- name: Set an environment variable using a multi-line string
run: |
echo "MULTI_LINE_STRING<<EOF" >> $GITHUB_ENV
echo "
Hello World!
Here's a
multi-line string.
" >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV
- name: Check the environment variable from the previous step
run: echo $MULTI_LINE_STRING
- name: Set build environment based on Git branch name
if: github.ref == 'refs/heads/demo' || contains(env.APP_NAME, 'demo')
run: echo "BUILD_ENV=demo" >> $GITHUB_ENV
- name: Use the GitHub Actions format function to provide some details about Spam
run: |
echo "SPAM_STRING=${{ format(
'Spam is short for {0} and is made from {1} by {2}',
'spiced ham',
'pork shoulder',
'Hormel'
) }}" >> $GITHUB_ENV
- name: Run a multi-line shell script block
run: |
echo "
Hello World, from ${{ env.APP_NAME }}!
Add other actions to build,
test, and deploy your project.
"
if [ "$BUILD_ENV" = "demo" ] || ${{ contains(env.APP_NAME, 'demo') }}; then
echo "This is a demo."
elif [ "$BUILD_ENV" ]; then
echo "BUILD_ENV=$BUILD_ENV"
else
echo "There isn't a BUILD_ENV variable set."
fi
echo "Did you know that $SPAM_STRING?"
- uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Run a multi-line Python script block
shell: python
run: |
import os
import sys
version = f"{sys.version_info.major}.{sys.version_info.minor}"
print(f"Hello World, from Python {version} and ${{ env.APP_NAME }}!")
print(f"Did you know that {os.getenv('SPAM_STRING', 'there is a SPAM_STRING')}?")
- name: Run an external shell script
working-directory: ./.github/workflows
run: . github-actions-workflow-demo.sh
- name: Run an external Python script
working-directory: ./.github/workflows
run: python github-actions-workflow-demo.py
name: demo
on:
push:
branches: [master, develop]
pull_request:
workflow_dispatch:
env:
APP_NAME: "GitHub Actions sample workflow with build matrix"
jobs:
matrix:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [macOS-latest, ubuntu-latest, windows-latest]
python-version: [3.6, 3.7, 3.8]
silly-word: [foo, bar, baz]
steps:
- uses: actions/checkout@v2
- name: Echo a silly word
run: echo ${{ matrix.silly-word }}
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Run a multi-line Python script block
shell: python
run: |
import os
import sys
version = f"{sys.version_info.major}.{sys.version_info.minor}"
print(f"Hello World, from Python {version}, ${{ matrix.os }}, and ${{ matrix.silly-word }}!")
GitHub Actions provides output like this:
You can see a demo workflow in br3ndonland/algorithms.
- Steps in a job are sequential by default.
- Jobs are parallelized by default, unless you control the order by using
needs
. - Action inputs and outputs: If you're unclear on what you can do with an action, navigate to the GitHub repo for the action and look for a file called action.yml, like this one in actions/checkout. This file is a manifest declaring what the action can do.
- Debugging: If you want more debugging information, add
ACTIONS_STEP_DEBUG
to your secrets parameter store.- Key:
ACTIONS_STEP_DEBUG
- Value:
true
- Key:
Secrets is an encrypted parameter store (key:value store). The syntax is similar to environment variables.
- GitHub Actions can use secrets, so you don't have to hard-code API keys and other credentials. Secrets are redacted from GitHub Actions logs.
- Each repo has a secrets store at Settings -> Secrets (
https://github.com/org/repo/settings/secrets
) - Each organization also has a secrets store that can be used in all the organization's repos.
- Service containers can be set up for test databases and other needs.
- Users can create their own actions. For example, container actions can run in Docker containers of your choosing.
-
VM info: The GitHub Actions runner provisions virtual machines with similar resources as AWS EC2
t2.large
instances.- 2-core CPU
- 7 GB of RAM memory
- 14 GB of SSD disk space
-
GitHub Actions is free for open-source repos. Pricing for other repos only kicks in if you exceed the allotted build minutes.
Poetry is a useful tool for Python packaging and dependency management. The following set of tips was originally posted to python-poetry/poetry#366.
Use actions/cache with a variation on their pip
cache example to cache Poetry dependencies for faster installation.
- name: Set up Poetry cache for Python dependencies
uses: actions/cache@v2
if: startsWith(runner.os, 'Linux')
with:
path: ~/.cache/pypoetry
key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
restore-keys: ${{ runner.os }}-poetry-
Installing Poetry via pip
can lead to dependency conflicts, so the custom installer is recommended. The command listed in the docs exits in GitHub Actions with 127
(not on $PATH
).
There are some additional modifications needed for GitHub Actions:
- Add
-y
to avoid prompts. - Add Poetry to
$GITHUB_PATH
(note that the::set-env
syntax has been deprecated). - Move
poetry install
to separate step to ensure Poetry is on$GITHUB_PATH
.
- name: Install Poetry
run: |
curl -fsS -o get-poetry.py https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py
python get-poetry.py -y
echo "$HOME/.poetry/bin" >> $GITHUB_PATH
- name: Install dependencies
run: poetry install --no-interaction
Poetry allows config from the poetry config
command, or by environment variables. Environment variables are a more dependable way to configure Poetry in CI.
env:
POETRY_VIRTUALENVS_CREATE: false
- Create a PyPI token.
- Add it to the GitHub Secrets store for the repo (Settings -> Secrets).
- Use the secret in your workflow with
${{ secrets.PYPI_TOKEN }}
(secret name isPYPI_TOKEN
in this example, and username for PyPI tokens is__token__
). - Use
poetry publish --build
to build and publish in one step.
- name: Build Python package and publish to PyPI
if: startsWith(github.ref, 'refs/tags/')
run: poetry publish --build -u __token__ -p ${{ secrets.PYPI_TOKEN }}
That's why they call it Poetry. Beautiful.
Expand this details element for an example workflow from br3ndonland/inboard that uses these tips.
name: builds
on:
push:
branches: [develop, master]
tags:
- "[0-9v]+.[0-9]+.[0-9a-z]+"
workflow_dispatch:
jobs:
python:
runs-on: ubuntu-latest
env:
POETRY_VIRTUALENVS_CREATE: false
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Set up Poetry cache for Python dependencies
uses: actions/cache@v2
if: startsWith(runner.os, 'Linux')
with:
path: ~/.cache/pypoetry
key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
restore-keys: ${{ runner.os }}-poetry-
- name: Set up pre-commit cache
uses: actions/cache@v2
if: startsWith(runner.os, 'Linux')
with:
path: ~/.cache/pre-commit
key: ${{ runner.os }}-pre-commit-${{ hashFiles('.pre-commit-config.yaml') }}
restore-keys: ${{ runner.os }}-pre-commit-
- name: Install Poetry
run: |
curl -fsS -o get-poetry.py https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py
python get-poetry.py -y
echo "$HOME/.poetry/bin" >> $GITHUB_PATH
- name: Install dependencies
run: poetry install --no-interaction -E fastapi
- name: Run pre-commit hooks
run: pre-commit run --all-files
- name: Run unit tests
run: pytest
- name: Build Python package and publish to PyPI
if: startsWith(github.ref, 'refs/tags/')
run: poetry publish --build -u __token__ -p ${{ secrets.PYPI_TOKEN }}
docker:
runs-on: ubuntu-latest
needs: [python]
steps:
- uses: actions/checkout@v2
- name: Log in to Docker registry
run: docker login ghcr.io -u ${{ github.actor }} -p ${{ secrets.PAT_GHCR }}
- name: Build Docker images
run: |
docker build . --rm --target base -t ghcr.io/br3ndonland/inboard:base --cache-from python:3.8
docker build . --rm --target starlette -t ghcr.io/br3ndonland/inboard:starlette
docker build . --rm --target fastapi -t ghcr.io/br3ndonland/inboard:fastapi
- name: Push Docker images to registry
run: |
docker push ghcr.io/br3ndonland/inboard:base
docker push ghcr.io/br3ndonland/inboard:starlette
docker push ghcr.io/br3ndonland/inboard:fastapi
- name: Add Git tag to Docker images
if: startsWith(github.ref, 'refs/tags/')
run: |
GIT_TAG=$(echo ${{ github.ref }} | cut -d / -f 3)
docker tag ghcr.io/br3ndonland/inboard:base ghcr.io/br3ndonland/inboard:base-"$GIT_TAG"
docker tag ghcr.io/br3ndonland/inboard:starlette ghcr.io/br3ndonland/inboard:starlette-"$GIT_TAG"
docker tag ghcr.io/br3ndonland/inboard:fastapi ghcr.io/br3ndonland/inboard:fastapi-"$GIT_TAG"
docker push ghcr.io/br3ndonland/inboard:base-"$GIT_TAG"
docker push ghcr.io/br3ndonland/inboard:starlette-"$GIT_TAG"
docker push ghcr.io/br3ndonland/inboard:fastapi-"$GIT_TAG"
- name: Tag and push latest image
run: |
docker tag ghcr.io/br3ndonland/inboard:fastapi ghcr.io/br3ndonland/inboard:latest
docker push ghcr.io/br3ndonland/inboard:latest
Dependabot now offers automated version updates, with (preliminary) support for Poetry 🎉. If you have access to the Dependabot beta, set up .github/dependabot.yml as described in the docs:
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"
Dependabot will now send you PRs when dependency updates are available. Although package-ecosystem
must be set to pip
, it will pick up the pyproject.toml and poetry.lock. Check the status of the repo at Insights -> Dependency graph -> Dependabot.
There's no PR merge trigger. There are two ways to implement this:
-
Use
pull_request:
types: [closed]
as the event trigger and only run each job if the event payload containsmerged == 'true'
. See the context and expression syntax docs for examples of how to use the event payload.name: demo on: pull_request: types: [closed] # release: # types: [published] jobs: job1: # if: github.event.pull_request.merged == 'true' || github.event.release.draft == 'false' if: github.event.pull_request.merged == 'true' steps: - name: step1 run: echo 'hello world'
-
Set up a
repository_dispatch
orworkflow_dispatch
webhook and use that as the trigger. The method for this is unclear, andworkflow_dispatch
events may only be read on the default branch.
Inability to set environment variables per-trigger: It's difficult to set environment variables per-trigger, such as based on which branch was checked out. There's a top-level env:
key, but it doesn't allow expressions or separate steps:
.
env: # Can't do this
- name: Set build environment to production if master branch is checked out
if: contains(github.ref, 'master')
run: echo "BUILD_ENV=production" >> $GITHUB_ENV
- name: Set build environment to development if develop branch is checked out
if: contains(github.ref, 'develop')
run: echo "BUILD_ENV=development" >> $GITHUB_ENV
- name: Set build environment to test otherwise
if: ${{ !contains(github.ref, 'master') || !contains(github.ref, 'develop') }}
run: echo "BUILD_ENV=test" >> $GITHUB_ENV
Furthermore, environment variables set to $GITHUB_ENV
within a job are scoped to that job.
The solution is to use outputs instead of environment variables. Set outputs from one job and read them in downstream jobs. Note that step outputs must also be set as job outputs in order to be passed to other workflows.
- The
needs:
feature is great, but I should be able to depend on other workflows, rather than having all the workflows in one file. workflow_dispatch
: seems like it could be what I'm looking for, but it's marketed as a way to manually trigger workflows in the UI, not as a way to automatically chain workflows together.- There's a new
workflow_run
trigger that may help to chain workflows together.
-
If a PR has merge conflicts, GitHub Actions workflows may not run at all. See this GitHub community thread.
-
Try using equality operators (
==
and!=
) to check out the PRHEAD
commit, instead of the default (the merge commit), as described in the actions/checkout README:- uses: actions/checkout@v2 if: ${{ github.event_name != 'pull_request' }} - uses: actions/checkout@v2 if: ${{ github.event_name == 'pull_request' }} with: ref: ${{ github.event.pull_request.head.sha }}
-
If checking out the
HEAD
commit doesn't work, you may need to resolve merge conflicts to continue.
-
In which context can I use which things?
- Where can I define
env:
? - Where can I define
defaults:
? - Where can I add
if:
? - When do I need
${{ }}
for expressions? The docs say, "When you use expressions in an if conditional, you may omit the expression syntax (${{ }}
) because GitHub automatically evaluates the if conditional as an expression, unless the expression contains any operators. If the expression contains any operators, the expression must be contained within${{ }}
to explicitly mark it for evaluation." However,if:
conditionals with expressions seem to work fine without${{ }}
.
- Where can I define
-
Object filters seem like a useful corollary to CloudFormation mappings, but there's no explanation for how to set up object filters within workflows.
-
There's no concept of
if/elif/else
. -
Syntactic subtleties, such as the requirement for single quotes in some YAML fields:
jobs: base: runs-on: ubuntu-latest steps: # this works - name: Set build environment to production if: github.ref == 'refs/heads/master' run: echo "BUILD_ENV=production" >> $GITHUB_ENV # this doesn't - name: Set build environment to production if: github.ref == "refs/heads/master" run: echo "BUILD_ENV=production" >> $GITHUB_ENV
-
github.ref
vsgithub.base_ref
: Confusingly,github.base_ref
appears to output different syntax thangithub.ref
.github.base_ref
returns just the branch name, likedevelop
.github.ref
returns the full Git ref, likerefs/heads/develop
.
- Overview
- Workflow triggers (the
on:
section of YAML workflow files): - Workflow syntax
- GitHub docs: Actions - Workflow syntax reference
- GitHub docs: Actions - Context and expression syntax (syntax in double curly braces like
${{ github.repository }}
, and functions likeformat()
) - GitHub docs: Actions - Workflow commands for GitHub Actions (shell commands you can perform within workflows, like
echo ::set-env name=COLOR::green
)
- GitHub Actions + GitHub Container Registry
- Actions
- GitHub Actions Marketplace
- actions/runner: the runner for GitHub Actions
- sdras/awesome-actions
- A Gist is actually a repository.
- To clone the Gist locally:
git clone [email protected]:f9c753eb27381f97336aa21b8d932be6.git github-actions
- No subdirectories are allowed.
- To add images to a Markdown file in a Gist:
- Commit the image file (in the same directory, no sub-directories)
- Push the change to GitHub with
git push
- Click on the "raw" button next to the image
- Copy the URL
- Add the URL to an image tag in the Markdown file.