Last active
July 11, 2019 16:47
-
-
Save mcg1969/cbb1760cea6b0671959d8cbf957c89bf to your computer and use it in GitHub Desktop.
Cached Miniconda in CircleCI
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
machine: | |
environment: | |
MINICONDA: "$HOME/miniconda" | |
MINICONDA_PATH: "$MINICONDA/bin" | |
PATH: "$MINICONDA_PATH:$PATH" | |
CONDA: "$MINICONDA_PATH/conda" | |
ANACONDA: "$MINICONDA_PATH/anaconda" | |
# This ensures we don't accidentally pull in ~/.condarc | |
# if it happens to be there for some strange reason | |
CONDARC: "$MINICONDA/condarc.none" | |
checkout: | |
post: | |
- git fetch --depth 1000000 | |
- git fetch --tags | |
dependencies: | |
override: | |
- rm -rf virtualenvs venv .pyenv | |
- | | |
if [[ ! -f $MINICONDA/.finished ]]; then | |
rm -rf $MINICONDA | |
wget $ANACONDA_REPO_URL/static/extras/Miniconda3.sh | |
bash Miniconda3.sh -b -p $MINICONDA | |
# Use --system so the settings are placed in $MINICONDA | |
# Add any other conda settings you like here | |
$CONDA config --system --add channels conda-forge | |
# If you'd like to make sure conda is up to date | |
$CONDA install conda conda-build anaconda-client --yes | |
touch $MINICONDA/.finished | |
fi | |
# Install some packages in our _cache environment that we know we're going to | |
# need during the build process---say, your conda package's dependencies. Because | |
# CircleCI saves this cache _before_ the build process proper, there is no way | |
# to just let that build process populate the cache. So we install them into | |
# this environment---and then promptly remove them. I have some rather large | |
# packages in my particular use case so this helps me. | |
- $CONDA create -n cache ... --yes # add packages here | |
- $CONDA remove -n cache --all --yes | |
# Ideally, what I'd do is blow away the tarballs and leave the extracted packages. | |
# That would save the extraction cost during the build process. Unfortunately, the | |
# absence of a tarball is interpreted as a missing package, because there's no MD5 | |
# signature available to verify. I understand why this is done (indeed I might | |
# have implemented it) but this is certainly a downside. So what I've chosen to | |
# do is clean out the extracted packages here and suffer the extraction cost, | |
# trading smaller cache tarball size for package extraction time. Experiments are | |
# needed to determine if it is worthwhile. | |
- $CONDA clean --packages --yes | |
# Likely unnecessary, but just in case. | |
- $CONDA build purge-all | |
cache_directories: | |
- ~/miniconda | |
test: | |
override: | |
- $CONDA build conda-recipe | |
deployment: | |
production: | |
branch: master | |
commands: | |
# The anaconda-client configuration is not cached so we have to set the URL here | |
- $ANACONDA config --set url $ANACONDA_REPO_URL/api | |
- $ANACONDA -t $ANACONDA_REPO_TOKEN upload --user creditflow --label dev --summary "auto-uploaded by circleci" $MINICONDA/conda-bld/*/*.tar.bz2 --force | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thanks @mcg1969, this came in handy for me today.