Skip to content

Instantly share code, notes, and snippets.

@mcg1969
Last active July 11, 2019 16:47
Show Gist options
  • Save mcg1969/cbb1760cea6b0671959d8cbf957c89bf to your computer and use it in GitHub Desktop.
Save mcg1969/cbb1760cea6b0671959d8cbf957c89bf to your computer and use it in GitHub Desktop.
Cached Miniconda in CircleCI
machine:
environment:
MINICONDA: "$HOME/miniconda"
MINICONDA_PATH: "$MINICONDA/bin"
PATH: "$MINICONDA_PATH:$PATH"
CONDA: "$MINICONDA_PATH/conda"
ANACONDA: "$MINICONDA_PATH/anaconda"
# This ensures we don't accidentally pull in ~/.condarc
# if it happens to be there for some strange reason
CONDARC: "$MINICONDA/condarc.none"
checkout:
post:
- git fetch --depth 1000000
- git fetch --tags
dependencies:
override:
- rm -rf virtualenvs venv .pyenv
- |
if [[ ! -f $MINICONDA/.finished ]]; then
rm -rf $MINICONDA
wget $ANACONDA_REPO_URL/static/extras/Miniconda3.sh
bash Miniconda3.sh -b -p $MINICONDA
# Use --system so the settings are placed in $MINICONDA
# Add any other conda settings you like here
$CONDA config --system --add channels conda-forge
# If you'd like to make sure conda is up to date
$CONDA install conda conda-build anaconda-client --yes
touch $MINICONDA/.finished
fi
# Install some packages in our _cache environment that we know we're going to
# need during the build process---say, your conda package's dependencies. Because
# CircleCI saves this cache _before_ the build process proper, there is no way
# to just let that build process populate the cache. So we install them into
# this environment---and then promptly remove them. I have some rather large
# packages in my particular use case so this helps me.
- $CONDA create -n cache ... --yes # add packages here
- $CONDA remove -n cache --all --yes
# Ideally, what I'd do is blow away the tarballs and leave the extracted packages.
# That would save the extraction cost during the build process. Unfortunately, the
# absence of a tarball is interpreted as a missing package, because there's no MD5
# signature available to verify. I understand why this is done (indeed I might
# have implemented it) but this is certainly a downside. So what I've chosen to
# do is clean out the extracted packages here and suffer the extraction cost,
# trading smaller cache tarball size for package extraction time. Experiments are
# needed to determine if it is worthwhile.
- $CONDA clean --packages --yes
# Likely unnecessary, but just in case.
- $CONDA build purge-all
cache_directories:
- ~/miniconda
test:
override:
- $CONDA build conda-recipe
deployment:
production:
branch: master
commands:
# The anaconda-client configuration is not cached so we have to set the URL here
- $ANACONDA config --set url $ANACONDA_REPO_URL/api
- $ANACONDA -t $ANACONDA_REPO_TOKEN upload --user creditflow --label dev --summary "auto-uploaded by circleci" $MINICONDA/conda-bld/*/*.tar.bz2 --force
@msarahan
Copy link

msarahan commented Feb 4, 2018

Thanks @mcg1969, this came in handy for me today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment