Auto-deploying built products to gh-pages with Travis

0-github-actions.md

Auto-deploying built products to gh-pages with GitHub Actions

This is a set up for projects which want to check in only their source files, but have their gh-pages branch automatically updated with some compiled output every time they push.

A file below this one contains the steps for doing this with Travis CI. However, these days I recommend GitHub Actions, for the following reasons:

• It is much easier and requires less steps, because you are already authenticated with GitHub, so you don't need to share secret keys across services like you do when coordinate Travis CI and GitHub.
• It is free, with no quotas.
• Anecdotally, builds are much faster with GitHub Actions than with Travis CI, especially in terms of time spent waiting for a builder.

Set up a build script

Set up your repository with a build script. This could be a checked-in build.sh, or a make command (I usually use make ci), or whatever.

Ensure that the build script outputs all the results to an out/ directory. You'll probably want to update .gitignore to include out/.

The build file

Add this file to .github/workflows/build.yml:

name: Build
on:
  pull_request:
    branches:
    - master
  push:
    branches:
    - master
jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Build
      run: make ci
    - name: Deploy
      if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
      uses: peaceiris/actions-gh-pages@v3
      with:
        github_token: ${{ secrets.GITHUB_TOKEN }}
        publish_dir: ./out

Here, where it says run: make ci, replace it with whatever build script you have, e.g. run: bash ./build.sh.

Similarly, if you are using a different branch name than master, e.g. if you are using main, then update the three locations which reference master.

Security considerations

This relies on third-party code in peaceiris/actions-gh-pages; this code is not maintained by GitHub. In theory, the @peaceiris user could update their action code to extract your GITHUB_TOKEN, and update their v3 tag to point to this new malicious commit. Read more about this in GitHub's docs

The best security against this, if you are concerned, is to pin to a specific commit, e.g. by replacing peaceiris/actions-gh-pages@v3 with peaceiris/actions-gh-pages@bbdfb200618d235585ad98e965f4aafc39b4c501 (which is the commit for what is currently tagged as v3.7.3). But, this of course means you'll fail to get updates, perhaps even security updates. So on balance, I'm currently recommending using the v3 tag. Your preferences may vary.

Example project

A recent project I maintain which uses this approach is WICG/import-maps. Some features if it you may enjoy perusing:

• The Makefile, especially if you're interested in building specifications with Bikeshed
• The .github/workflows/test.yml file, which shows how to run other build actions alongside the GitHub pages build action in build.yml.

auto-deploy.md

Auto-deploying built products to gh-pages with Travis

This is a set up for projects which want to check in only their source files, but have their gh-pages branch automatically updated with some compiled output every time they push.

Create a compile script

You want a script that does a local compile to e.g. an out/ directory. Let's call this compile.sh for our purposes, but for your project it might be npm build or gulp make-docs or make or anything similar.

The out/ directory should contain everything you want deployed to gh-pages. That almost always includes an index.html.

Check this script in to your project.

Sign up for Travis and add your project

Get an account at https://travis-ci.com/. Turn on Travis for your repository in question, using the Travis control panel.

Get encrypted credentials

The trickiest part of all this is that you want to give Travis the ability to run your deploy script and push changes to gh-pages, without checking in the necessary credentials to your repo. The solution for this is to use Travis's encrypted file support.

NOTE: an earlier version of this guide recommended generating a GitHub personal access token and encrypting that. Although this is simpler, it is not a good idea in general, since it means any of your repository's collaborators would be able to edit the Travis build script to email them your access token, thus giving them access to all your repositories. The repository-specific deploy key approach is safer.

First, generate a new SSH key. You should not reuse existing SSH keys, and you should not add the SSH key to your GitHub account. Also, you must ensure that you do not include a passphrase (i.e., just press enter when asked for one).

Next, add that deploy key to your repository at https://github.com/<your name>/<your repo>/settings/keys.

Now use the Travis client to encrypt the generated deploy key. The result should look something like this:

$ travis encrypt-file deploy_key
encrypting deploy_key for domenic/travis-encrypt-file-example
storing result as deploy_key.enc
storing secure env variables for decryption

Please add the following to your build script (before_install stage in your .travis.yml, for instance):

    openssl aes-256-cbc -K $encrypted_0a6446eb3ae3_key -iv $encrypted_0a6446eb3ae3_key -in deploy_key.enc -out deploy_key -d

Pro Tip: You can add it automatically by running with --add.

Make sure to add deploy_key.enc to the git repository.
Make sure not to add deploy_key to the git repository.
Commit all changes to your .travis.yml.

You should follow the instructions and commit deploy_key.enc to the repository. You should also add deploy_key to your .gitignore, or delete it. Ignore the bits about .travis.yml, however; we're going to do that part all custom-like.

Create your .travis.yml file

With all this in hand, you can create a .travis.yml file. It should look like this:

language: generic # don't install any environment
script:
- bash ./compile.sh

branches:
  only:
  - master

before_deploy:
- openssl aes-256-cbc -K $encrypted_0a6446eb3ae3_key -iv $encrypted_0a6446eb3ae3_key -in deploy_key.enc -out deploy_key -d


deploy:
  provider: pages
  local_dir: out
  deploy_key: deploy_key
  edge:
    branch: master

notifications:
  email:
    on_success: never
    on_failure: always

If your compile script depends on certain environment features, you might want to set up the environment using Travis's built-in abilities, e.g. by changing the language lines like so:

language: node_js
node_js:
- stable

(In this case, by default Travis will install the latest stable Node.js, then run npm install.)

Finishing up

At this point you should have 3 files checked in to your repo: compile.sh deploy_key.enc, and .travis.yml. If you've also told Travis about your repo, then the first time you push to GitHub with these changes, it will automatically compile and deploy your source!

See it in action

I use basically this exact setup for https://github.com/wicg/origin-policy. The relevant files are:

• https://github.com/WICG/origin-policy/blob/master/.travis.yml
• https://github.com/WICG/origin-policy/blob/master/Makefile (so I could use make ci instead of bash ./compile.sh)
• https://github.com/WICG/origin-policy/blob/master/deploy_key.enc

Licensing

All code in the above post is licensed under the MIT License.

As @dspinellis pointed out I needed to use git diff --quiet instead of git diff --exit-code. In my case because the --exit-code passes the entire diff output to bash and bash choked on it.

@stefanbc you can chmod u+x deploy.sh in your local repo then add and commit deploy.sh to the repo. Git tracks permissions (or at least the executable bit)

I think a better way to clean out the existing contents is to switch into the repository and do this

# Clean out existing contents
git ls-files | xargs rm -rf

Also, I think performing the following check before pushing (while not perfect) is probably a good idea

git fetch origin master

if [ "$(git rev-parse origin/master)" == "$SHA" ]; then
  git push "$SSH_REPO" "$TARGET_BRANCH"
else
    echo "The commit this build was started for is not the one on master. Doing nothing."
fi

@jilulu Have the same problem, asking for passphrase even though none was defined. I created the key using the git bash under Windows. As a test I also generated one using PuTTYgen, but same problem. Maybe this is related: https://ubuntuforums.org/showthread.php?t=1828015

EDIT: It seems as if the encryption (or decryption) of the private key produces garbage. When I output the key content while running in Travis CI after decrypting I get just nonsense.

@jilulu @letmaik @domenic, @RyanNHG, @wings27 and @ShikherVerma I have generated ssh keys using the travis CLI to use it as a deployment keys for ssh, but my deployment script gets struct at ssh-add deploy_key ,due to Enter PassPhrase line ,I didnt keep any passphrase for the key but the ssh-add is asking for a password ,according other online sources the script should automatically run , why is my travis build getting struck. here is over all problem view in an image

Hey, I was having an error using this. I put all of my stuff in the description of the issue. It is right over here.

Good post, but contains some typo

- openssl aes-256-cbc -K $encrypted_0a6446eb3ae3_key -iv $encrypted_0a6446eb3ae3_key -in super_secret.txt.enc -out super_secret.txt -d
+ openssl aes-256-cbc -K $encrypted_0a6446eb3ae3_key -iv $encrypted_0a6446eb3ae3_iv -in super_secret.txt.enc -out super_secret.txt -d

hey I was having trouble with Enter Passphrase: too, even though I generated the key without any passphrase,
so I ended up with replacing

eval `ssh-agent -s`
ssh-add deploy_key

with

cp deploy_key ~/.ssh/id_rsa

and it seems to be working fine now

@domenic Recommend not writing private key without passphrase to disk. Something like this might be better:

eval `ssh-agent -s`
openssl aes-256-cbc -K $ENCRYPTED_KEY -iv $ENCRYPTED_IV -in deploy_key.enc -d | ssh-add -

Travis appears to have a deployment provider specifically for GitHub Pages now: https://docs.travis-ci.com/user/deployment/pages/

What if we don't want to overwrite older releases on GitHub pages? I'd like to keep hosting multiple compiled versions for testing.

When you're on windows, the travis encryption might not work. If you're using vagrant, you can simple add this Vagrantfile: https://gist.github.com/Argelbargel/022270d7ffdf7316e2811a7f7dbea81c to your github-project, set environment variables with your access-token and user-name and it will create and encrypt a new deploy_key

I had the problem with Enter Passphrase: too. In my case, I was using windows 10 so I tried with another OS, specifically Ubuntu. I followed the same steps and voilà!, It works... :)

Minor bug here:

chmod 600 ../deploy_key
eval `ssh-agent -s`
ssh-add deploy_key

Last line needs to be ssh-add ../deploy_key

Looks like the built-in Travis gh-pages deployment suffers from the same security problem described here:

NOTE: an earlier version of this guide recommended generating a GitHub personal access token and encrypting that. Although this is simpler, it is not a good idea in general, since it means any of your repository's collaborators would be able to edit the Travis build script to email them your access token, thus giving them access to all your repositories. The repository-specific deploy key approach is safer.

Looks like the built-in Travis gh-pages deployment suffers from the same security problem described here:

@RByers where have you found evidence of this?

Alternatively one could use

• GitHub username and personal access token, here demonstrated as GITHUB_USER and GITHUB_TOKEN

TOKEN_REPO=${REPO/github.com/$GITHUB_USER:$GITHUB_TOKEN@github.com}

# strip all the deploy key stuff and go straight to
git push $TOKEN_REPO $TARGET_BRANCH

An example can be found here

needs a

git add -N .

before the git diff for when the files are all new. (git diff doesn't diff new files otherwise)

Also ssh-add deploy_key should be ssh-add ../deploy_key I think

This script automates the configuration of the repo using a temporal docker container:

#!/bin/sh

apk add -U --no-cache git openssh ruby ruby-dev libffi-dev build-base ruby-dev libc-dev libffi-dev linux-headers
gem install travis --no-rdoc --no-ri

git clone $REPO ./tmp-repo && cd tmp-repo

git config user.name "Travis CI"
git config user.email "travis@gh-pages"

ssh-keygen -t rsa -b 4096 -C "travis@gh-pages" -f deploy_key
cat deploy_key.pub

travis login --org --auto
msg=$(travis encrypt-file deploy_key)

rm deploy_key
git add deploy_key.enc
git commit -m "Add deploy_key.enc `echo $msg | grep -o "encrypted_.*_key -iv" | sed -e 's/encrypted_\([0-9a-z]*\)_key -iv/\1/g'`"

sh

• Run $(command -v winpty) docker run --rm -it alpine sh -c "REPO='https://github.com/<user|organization>/<repo>'; $(cat travis-enc.sh)"
• Follow the steps until [master <SHORT_COMMIT_SHA>] Add deploy_key.enc <ENCRYPTION_LABEL> is shown.
• Rebase, squash, fixup, move commit to different branch...
• When you are OK with the result, git push.
• exit to close and remove the container.
• Check the log and copy the public key to the repo: https://developer.github.com/v3/guides/managing-deploy-keys/#deploy-keys

Tested, travis now takes care of all of that in one simple deploy step: https://docs.travis-ci.com/user/deployment/pages/

This

ENCRYPTED_KEY_VAR="encrypted_${ENCRYPTION_LABEL}_key"
ENCRYPTED_IV_VAR="encrypted_${ENCRYPTION_LABEL}_iv"
ENCRYPTED_KEY=${!ENCRYPTED_KEY_VAR}
ENCRYPTED_IV=${!ENCRYPTED_IV_VAR}
openssl aes-256-cbc -K $ENCRYPTED_KEY -iv $ENCRYPTED_IV -in ../deploy_key.enc -out ../deploy_key -d
chmod 600 ../deploy_key
eval `ssh-agent -s`
ssh-add deploy_key

can be written as follows:

eval `ssh-agent -s`
openssl aes-256-cbc -K `eval echo encrypted_$\{${ENCRYPTION_LABEL}_key\}` -iv `eval echo encrypted_$\{${ENCRYPTION_LABEL}_iv\}` -in deploy_key.enc -out deploy_key -d | ssh-add -

https://travis-ci.org/falsandtru/pjax-api/jobs/324621181#L3056 (with $ travis env set -P ENCRYPTION_LABEL encrypted_xxx)

Thanks @bfritz.

There is now this:
https://docs.travis-ci.com/user/deployment/pages/

Unfortunately, the solution described here doesn't works if you use Git LFS.
It returns the following error -

ERROR: Authentication error: Authentication required: You must have push access to verify locks
error: failed to push some refs to '[email protected]:username/reponame.git'

So, I had to go with old-fashion aproach with GitHub personal access token.

NOTE: an earlier version of this guide recommended generating a GitHub personal access token and encrypting that. Although this is simpler, it is not a good idea in general, since it means any of your repository's collaborators would be able to edit the Travis build script to email them your access token, thus giving them access to all your repositories. The repository-specific deploy key approach is safer.

Found a workaround for that by creating "machine user", as mentioned here https://docs.travis-ci.com/user/deployment/releases/ -

For security, it’s ideal for api_key to have write access limited to only respositories where Travis deploys to GitHub releases. The suggested workaround is to create a machine user — a dummy GitHub account that is granted write access on a per repository basis.