Automate build workflow for Homebrew tap bottles (Linux and macOS)

How-to-automate-build-bottles-your-homebrew-tap.md

How to automate the build of bottles on your Homebrew tap

Note on Oct 4, 2018: due to a change in Homebrew's brew test-bot behaviour, the user must set HOMEBREW_TRAVIS_CI and HOMEBREW_TRAVIS_SUDO appropriately (it was previously using Travis-CI-provided TRAVIS and TRAVIS_SUDO).

This tutorial is a follow-up to the discussion we had on davidchall/homebrew-hep#114. It relies on a fork of the test-bot provided by davidchall; you can get it with brew tap maelvalais/test-bot. First:

1. the Github project must be of the form https://github.com/<user>/homebrew-<tap> with the following tree (I give the example of one of my formulas, touist):

   .
   ├── touist.rb
   └── .travis.yml
   

2. the Bintray project must be of the form https://bintray.com/<user>/bottles-<tap>

3. the S3 bucket can have any name

Use Travis CI to test and bottle your formulae

The idea is to use staged build in Travis CI, where the two stages will be

1. the testing stage where the bottles are built and tested for the different architectures with

   brew test-bot
   

   At the end of this stage, each job must upload its bottle.tar.gz and bottle.json to somewhere (e.g., free AWS S3 bucket).

2. the deploy stage (only on master) where we fetch the bottle.json and bottle.tar.gz, upload the bottle.tar.gz to Bintray and merge the multiple bottle.json into a commit that contains the bottle DSL with the command

   brew test-bot --ci-upload
   

You can see a detailed example of such a configuration in the .travis.yml (this gist), where bottles are built for linux, sierra and el_capitan. You can also see an example of tap, touist/homebrew-touist, that uses it and its .travis.yml (homebrew-touist repo).

Workflow for updating formulas

The Homebrew/homebrew-core has the following workflow:

Instead of pushing pr-1234 (containing the updated DSL commit) to a fork (in core's case, the fork is BrewTestBot/homebrew-core), I propose to use the same repo. Don't worry, the pr-1234 tags won't show in people's clones. Here is the workflow:

Drawings made using Lucidchart (not free).

When a PR is opened and the formula needs a bottle, Travis CI will build the bottle (brew test-bot) and then run brew test-bot --ci-upload which will:

• upload the bottle as "unpublished".
• commit and push the DSL to BrewTestBot/Homebrew-core with the pr-1234 tag.

Then a Homebrew maintainer (me) will do

brew pull --bottle --bintray-org=touist --test-bot-user=touist https://github.com/touist/homebrew-touist/pull/5

which will do two things:

• switch the bottle in bintray from "unpublished" to "published"
• fetch the pr-1234 tag, rebase it on top of Homebrew/homebrew-core

then the mainainer push the merged PR to Homebrew/homebrew-core.

Drawbacks w.r.t. bottles when maintaining a tap relying on core formulae

On the homebrew-core repo, whenever a formula on which some other formulae are relying on is updated, we also must rebuild the bottles of these dependent formulae (this is why we have the revision number _1 after the actual version number). This seems to be automatically taken care of on the homebrew-core tap.

On your own tap, this revision bump whenever a dependency bottle gets rebuilt can lead to broken bottles. This problem has not been solved yet (see Homebrew/brew#3346 and Homebrew/brew#2572). But we can:

1. run a cron job that tests the bottles every night to check   if the bottle still works, using brew test *.rb and brew linkage --test *.rb;
2. or we could subscribe to the changes on the formulae we depend on and run a script on a Travis CI build (I could not find a way to do that yet).

What you need

1. the name of the Github repo for your tap must be <user>/homebrew-<tap>
2. you need a Github OAuth2 token that you can get in your Github settings   (secret variable GITHUB_TOKEN in .travis.yml);
3. the name of the Bintray repo must be <bintray-user>/bottles-<tap>   (secret variable HOMEBREW_BINTRAY_KEY in .travis.yml);
4. you also need an AWS S3 bucket, it is free and takes 1 minute to create; then you need to create a token+password and set them as secret variables in   the travis-ci settings (secret variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in .travis.yml).

Tricks I want to remember when I was setting up Travis CI + test-bot

1. when using brew test-bot, never use --ci-pr, --ci-master, --ci-testing on your mac as it is going to remove everything installed. You can still use brew test-bot and brew test-bot --ci-upload; what I would do is to make sure everything runs correctly locally (except for brew doctor which generally always fails on my mac) using:

   brew test-bot touist.rb
   

2. On Travis CI, mac builds are costly (in time). For trying to make the   .travis.yml work, I really recommend to use os: linux instead of a mac image.

3. Also, when you find yourself stuck when trying to make the CI work, I recommend to re-run your job in debug mode in order to see what is going wrong. Again, as mac images are slow, I recommend to use the linux alternative.

4. if you see that brew test-bot is not building/testing any bottle, check   that you have made the symlink (in travis-ci) at $(brew --repo touist/touist)   that should target $TRAVIS_BUILD_DIR. It is important because brew test-bot   is testing the tap inside $(brew --repo) but the actual repo cloned by travis is in $TRAVIS_BUILD_DIR.

5. the --keep-old option when uploading seems to be a good idea (= keeps bottles you previously built but that you don't produce anymore) but it keeps failing on stupid errors all the time; I don't use it anymore.

6. adding "revision 1" to a formula does not seem to be appropriate for forcing a bottle to be rebuild

7. the tests/building of the bottle only happens when you push a series of commit on ONE single formula; if you include some edits to .travis.yml for example, the push won't trigger build of the bottle. This caused me a lot of time lost on try and errors.

8. when creating a new formula, you may get these kind of errors during the build (on PR, branch and master):

   * New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
   * GitHub fork (not canonical repository)
   * GitHub repository too new (<30 days old)
   

  First, make sure the bottles build without errors and that the only step failing is brew audit --online. Then, accept the PR and make a small change to the formula to force the rebuild of the bottle. The errors will disappear (as they are only there when the formula is new).

Note on wierd prefix: and cellar: in bottle DSL

WARNING: in my solution, in the deploy state, I do on a linux build:

• brew test-bot --ci-upload

which packs all bottles at the same time and I get weird prefix: /usr/local and cellar: :any_skip_relocation. I fixed this with a little trick: change every /usr/local with /home/linuxbrew/.linuxbrew so that brew test-bot --ci-upload removes the cellar: field when it is the default cellar.

Example of such problem: MacOS lingeling-151109.el_capitan.bottle.json and lingeling-151109.sierra.bottle.json have:

{
    "prefix": "/usr/local",
    "cellar": "any_skip_relocation",
}

but the linux bottle lingeling-151109.x86_64_linux.bottle.json has:

{
    "prefix": "/home/linuxbrew/.linuxbrew",
    "cellar": "/home/linuxbrew/.linuxbrew/Cellar",
}

and it wrongly became the following bottle DSL, probably because the linux bottle.json is the last to be read, so it takes precedence over the previous ones:

  bottle do
    root_url "https://dl.bintray.com/touist/bottles-touist"
    sha256 "141f132d5ed58f930ada22fbe81d63e4115b9f78b8fb2ca3fa266e30e5fdf0a3" => :sierra
    sha256 "be1e33155185e36cbfbc48a743efd14d65edbfc174d2e31e4386414ea2153233" => :el_capitan
    sha256 "b24481b3ac4a7b02114ea5d627aff7cd6e2bb1551e4b4f073d3aa1d580f787c8" => :x86_64_linux
  end

It should have been

  bottle do
    root_url "https://dl.bintray.com/touist/bottles-touist"
    cellar :any_skip_relocation if OS.mac?
    sha256 "141f132d5ed58f930ada22fbe81d63e4115b9f78b8fb2ca3fa266e30e5fdf0a3" => :sierra
    sha256 "be1e33155185e36cbfbc48a743efd14d65edbfc174d2e31e4386414ea2153233" => :el_capitan
    sha256 "b24481b3ac4a7b02114ea5d627aff7cd6e2bb1551e4b4f073d3aa1d580f787c8" => :x86_64_linux
  end

Notes on my discoveries of Linuxbrew

• When a formula is linuxbrew-only, it has the comment # tag "linuxbrew" in it.

What to do when some dependencies are updated?

As mentionned by Steven Peters here, when a dependency is updated, we must also update the formula that depends on it. In 'core', this is done by incrementing the revision number for all formulas that rely on the updated formula.

In this tap, a cron is daily testing every formula (brew install <formula> && brew linkage --test <formula>). If a linkage breakage is detected, it automatically increments the revision number. It has been working fine for now, the only problem is to have travis-ci working every single day (I am getting better and better at that).

travis.yml

# Inspired from the .travis.yml in
# https://github.com/davidchall/homebrew-hep/

# This script can be executed in three different modes:
# - cron/api (api is when you select 'trigger build' in travis): in this mode,
#   all the formulas are tested to make sure that there is no shared lib
#   breakage
# - push to master: does nothing
# - pull-request: in this mode, the updated formulas are built and bottled
#   and the bottles are pushed to bintray as 'unpublished'. The DSL commit
#   is pushed as a tag, e.g., 'pr-1234', which us unrelated to master.

language: ruby
if: tag IS blank AND NOT type IN (push)
env:
  global:
    - HOMEBREW_LOGS=/tmp # Prevent ~/.cache/Homebrew/Logs from being rebuilt
    - HOMEBREW_NO_AUTO_UPDATE=yes # Prevents redundant auto-updates from brew

    ###### Bintray configuration ######
    - TAP_BOTTLE_ORG=touist                # Used in this script only
    - TAP_BOTTLE_ROOT_URL=https://dl.bintray.com/$TAP_BOTTLE_ORG/bottles-touist
    - HOMEBREW_BINTRAY_USER=maelvalais
    # HOMEBREW_BINTRAY_KEY                 # Secret variable set in Travis CI

    ###### AWS S3 configuration ######
    - AWS_REGION=eu-west-1                 # aws-sdk-s3 gem errors without it
    - BUCKET=homebrew-touist-travis
    # AWS_ACCESS_KEY_ID                    # Secret variable set in Travis CI
    # AWS_SECRET_ACCESS_KEY                # Secret variable set in Travis CI

    ###### Github configuration ######
    # This config allows test-bot to push tags (eg. pr-1234) and DSL commits.
    # They will be pushed to github.com/$GITHUB_USER/homebrew-%TAP%.
    - GITHUB_USER=touist
    # GITHUB_TOKEN                   # (Secret) For pushing bottle DSL commits

# (1) remember to use 'brew cleanup -s' at the end of the travis script;
# otherwise, the $HOME/Library/Caches/Homebrew folder will be enormous
# (~2GB for the 4 jobs). This is because this folder is already present on
# osx images and it is big.
# (2) For a time, I had removed $HOME/.cache/Homebrew (cache of Linuxbrew)
# because it was also storing some logs in $HOME/.cache/Homebrew/Logs, making
# the cache rebuilt every time.
cache:
  directories:
    - "$HOME/.cache/pip"
    - "$HOME/.gem/ruby"
    - "$HOME/Library/Caches/Homebrew" # (1)
    - "$HOME/.cache/Homebrew"         # (2)

install:
  # the official test-bot won't let you run inside TravisCI, so we use
  # davidchall's one. David's test-bot cannot push the commit using
  # Oauth github + https (only ssh) so I use my own.
  - brew tap maelvalais/test-bot
  # Install the ruby AWS gem so that I can upload bottles to S3
  - rvm default do gem install aws-sdk-s3
  # IMPORTANT STEP: link the tap inside brew to our current travis-cloned tap
  # Step: 1) create the intermediate folders <user>/<repo> so that
  # we can 2) remove <repo> and 3) replace it with a sym link
  # that will point to the travis build folder.
  # If we don't do that, the tap be cloned using the default master
  # branch, and thus we cannot test our tap at the current pushed commit.
  # We also need to unshallow in 4) because sometimes travis does not
  # clone thouroughly but we need a deep clone.
  - mkdir -p $(brew --repo $TRAVIS_REPO_SLUG)   # 1)
  - rm -rf $(brew --repo $TRAVIS_REPO_SLUG)     # 2)
  - ln -s $PWD $(brew --repo $TRAVIS_REPO_SLUG) # 3)
  - git fetch --unshallow || true               # 4)
  # 'brew doctor' must be run under HOMEBREW_DEVELOPER=1. Otherwise,
  # it returns 1 with the message "this osx version is outdated" on old osx ver.
  - HOMEBREW_DEVELOPER=1 brew doctor
  # Note on HOMEBREW_DEVELOPER: I don't want to put in env.global because
  # it should be '1' only during test-bot. If it is '1' during
  # brew cask uninstall... it will fail on the deprecation notice.
  # Unless I am in 'brew test-bot', I don't want to fail on warnings.

# WARNING: I discovered that 'set -e' is breaking Travis CI. I was using for
# stopping a multiple-lines command as soon as one of the commands fails.
# After having 'set -e', internal travis commands may stop working, e.g.:
#   /Users/travis/.travis/job_stages: line 57: shell_session_update: command not found
# is an internal error that shouldn't stop the script for continuing, but
# because of 'set -e', the whole internal Travis CI script stops.
# Conclusion: avoid using 'set -e' and more generally avoid for-loops and
# multi-line commands that should fail when one of the commands fails.
script:
  - |
    if [[ $TRAVIS_EVENT_TYPE =~ cron|api ]]; then
      brew install *.rb && brew test *.rb && brew linkage --test *.rb || exit 123
    fi
  - if [[ $TRAVIS_EVENT_TYPE =~ pull_request ]]; then brew test-bot --root-url=$TAP_BOTTLE_ROOT_URL; fi
  - ls *.bottle*.* || echo "==> No bottle created here"

jobs:
  include:
    - &run-osx
      os: osx
      osx_image: xcode9.4
      env: OS=high_sierra-10.13
      # We must use 'rvm: system' because the system ruby doesn't rely on
      # Homebrew, which allows us to reinstall Homebrew without having a
      # damaged ruby. We need ruby because we use the 'aws-sdk-s3' gem.
      rvm: system
      before_install: # IMPORTANT: HOMEBREW_DEVELOPER must not be set here.
        # First we uninstall any outdated versions of xquartz; otherwise,
        # Homebrew will complain of of older version (2.9.7) being outdated
        # even though we install a new version. Remember that
        # /usr/local/Caskroom will also be deleted below.
        - brew cask outdated xquartz || brew cask uninstall xquartz
        # Three reasons not to use the /usr/local and Homebrew installations
        # that come in the Travis CI images:
        # 1) because Travis CI has installed many non-homebrew things into
        #    /usr/local that randomly cause 'brew doctor' to fail;
        # 2) after time, the osx image contains an outdated Homebrew that
        #    has weird 'unlinked kegs' and such;
        # 3) also because it takes a long time to 'brew update' from an old
        #    Homebrew anyway, so why not start fresh.
        - mkdir ~/usr_local && sudo mv /usr/local/* ~/usr_local
        - /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

        # Either xquartz was already installed at its latest version or it has
        # been uninstalled. First, we put the cask back in place if it has not
        # been uninstalled.
        - |
          if [ -d ~/usr_local/Caskroom/xquartz ];
          then sudo mv ~/usr_local/Caskroom /usr/local/Caskroom;
          else travis_retry brew cask install xquartz; fi
        # We still need the Homebrew ruby on macOS 10.12 and 10.11 because the
        # system ruby uses an old openssl version ("tlsv1 alert protocol").
        - travis_retry brew install libyaml gmp [email protected] openssl

    - <<: *run-osx
      os: osx
      osx_image: xcode9.2
      env: OS=sierra-10.12

    - <<: *run-osx
      os: osx
      osx_image: xcode8
      env: OS=el_capitan-10.11

    - &run-on-linux
      os: linux
      env: OS=x86_64_linux
      before_install:
        # Fix the permission problem on linux (664 instead of 644) during
        # git clone (the one done by travis-ci). Homebrew needs formulas to be
        # 'chmod 644'. This is because git does not conserve permissions and
        # travis-ci seems to have by default a umask too permissive.
        # Because we cannot do 'umask 002' just before travis clones the repo,
        # I set umask afterwards (1) and I change the permission of
        # already cloned files from 664 to 644 (2).
        - umask 022          # (1)
        - chmod 0644 *.rb    # (2)

        # Instal linuxbrew
        - export PATH="/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:$PATH";
        # I added 'brew vendor-install ruby' because sometimes the install would
        # fail on 'Homebrew must be run under Ruby 2.3!' error.
        - yes | sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)" || (brew vendor-install ruby && brew update --force)
        # Fix a `brew doctor` error on "config" scripts for some reason
        - sudo rm -f /home/travis/.phpenv/shims/php-config
          /opt/pyenv/shims/*-config /usr/local/clang-*/bin/llvm-config

    # Uploading to bintray (as unpublished bottle) and pushing a tag to the
    # repo are only enabled on pull_requests.
    - <<: *run-on-linux
      stage: deploy
      if: branch = master AND type IN (pull_request)
      os: linux
      env: OS=any_linux
      script:
        # First step: download the *.json and *.tar.gz from the S3 bucket
        - |
          rvm default do ruby <<EOF || true
          require 'aws-sdk-s3'; s3 = Aws::S3::Resource.new
          s3.bucket("$BUCKET").objects(prefix:"$TRAVIS_BUILD_NUMBER/").each { |o| o.download_file(o.key.to_s.gsub(/^\d*\/(.*)/, '\1')) }

        # This hook will add [ci skip] the the commit messages for bottle DSL
        # so that travis-ci does not re-build a duplicate bottle on pushing.
        - echo 'echo "\n\n[ci skip]" >> "$1"' > .git/hooks/commit-msg; chmod +x .git/hooks/commit-msg

        # (1) We fail 'silently' if no bottle is found.
        # (2) A trick so that 'prefix' and 'cellar' are removed if they are
        # equal to DEFAULT_PREFIX and DEFAULT_CELLAR even though we run
        # 'brew bottle' on a linux setup.
        # (3) In 'brew --ci--upload', --git-name is the name of the fork
        # where the pr-1234 (or testing-98 when not in a PR) tag is pushed.
        # In my case, I use the 'main' instead of a fork.
        - |
          if ls *.{json,tar.gz}; then                                  # (1)
            sed -i 's:/usr/local:/home/linuxbrew/.linuxbrew:g' *.json; # (2)
            brew test-bot --ci-upload --root-url=$TAP_BOTTLE_ROOT_URL --git-name=$GITHUB_USER [email protected] --bintray-org=$TAP_BOTTLE_ORG --verbose;
          else
            echo "==> No bottle found in the bucket, skipping --ci-upload";
          fi
      after_script: true # nothing

after_script:
  # In case the AWS_SECRET_ACCESS_KEY and AWS_SECRET_ACCESS_KEY are not
  # available, we don't want the build to fail.
  # Here, the 'testing' jobs have finished and must upload their *.{json,tar.gz}
  - |
    rvm default do ruby <<EOF || true
    require 'aws-sdk-s3'; s3 = Aws::S3::Resource.new
    Dir["*.{json,tar.gz}"].each { |f| s3.bucket("$BUCKET").put_object(key:"$TRAVIS_BUILD_NUMBER/#{f}").upload_file(f) }

before_cache:
  # Scrub cache so that travis only caches stuff for installed formulae.
  - brew cleanup -s
  - brew cask cleanup
  # Remove temporary stuff (idk why they put that in ~/.cache... that should be in /tmp)
  - rm -f ~/Library/Caches/Homebrew/linkage.db ~/.cache/Homebrew/linkage.db
  # List the formulae so that I understand why the cache is huge sometimes
  - brew list
  - du -h ~/Library/Caches/Homebrew || du -h ~/.cache/Homebrew/linkage.db

Note on Oct 4, 2018: due to a change in Homebrew's brew test-bot behaviour, the user must set HOMEBREW_TRAVIS_CI and HOMEBREW_TRAVIS_SUDO appropriately (it was previously using Travis-CI-provided TRAVIS and TRAVIS_SUDO).

This is not the case. These are set by brew test-bot: https://github.com/Homebrew/homebrew-test-bot/blob/650adb8e413c3b902e87baa125c49f2e16760d5a/cmd/brew-test-bot.rb#L1558-L1565

run a cron job that tests the bottles every night to check if the bottle still works, using brew test *.rb and brew linkage --test *.rb;
or we could subscribe to the changes on the formulae we depend on and run a script on a Travis CI build (I could not find a way to do that yet).

Good idea. I'd recommend a Travis CI daily cron job for this.

you also need an AWS S3 bucket, it is free and takes 1 minute to create; then you need to create a token+password and set them as secret variables in the travis-ci settings (secret variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in .travis.yml).

Is it possible to upload only to Bintray?

---

As a general comment: nice work! It would be great to get this included in some form in Homebrew's official documentation and submit some of these issues and workarounds as issues or fixes in brew test-bot and other commands.

Nice work. I am prototyping a bintray-only variant of this for FreeCAD but brew test-bot is failing with Error: undefined local variable or method verbose. Thoughts?

Have you considered adding GitHub issues to your https://github.com/maelvalais/homebrew-test-bot fork so there is a way to file and resolve issues?

Same problem here, @bblacey.

@bblacey @izaid - I've put up a working example using Azure Pipelines, it might be helpful for you :)

https://github.com/ladislas/homebrew-greetings

Wow, excellent work, very well documented! 🎉 🎉 🎉
I'll move everything to Azure Pipelines asap.
Thank you @ladislas 👍

@ladislas, @maelvalais, I managed to get things working reasonably well using Bintary as a free intermediary but your Azure pipeline approach offers cleaner/separation and abstraction... Since my last stab at this, FreeCAD has started moving to Conda as a cross-platform build system because many of the APIs and extensions are python (think of FreeCAD the application as being a core CAD engine written in C++ with exposed Python Bindings so the rest of the app, plugins, extensions, etc are written in Python where Conda makes a lot of sense. Great work and thanks for letting me know. I am sure this will benefit the broader HomeBrew community because personal taps can be a huge time sink without auto-management.

@maelvalais my pleasure :)
by all means, if you find any issues or think that the documentation could be improved, please open an issue/pr!
your work here has been very helpful for me and helped me dig deep into brew test-bot. I could not use travis as my compilation time for avr-gcc was too long, but then I discovered Azure Pipelines and voila! :)

I really really have to move to Azure pipelines and away from Travis 🙂 I'm worried that sooner or later, the upstream brew test-bot will drop support for Travis, so I need to get moving

I think they already did... and Azure as well. Github Actions is the future. Haven't moved my workflows yet but I'm planning on it. I'll let you know how it goes.