venv-deployment.md

Howdy!

Alright, I think I found out the issue here. it's kind of annoying, I am not familiar enough with pip to know if there's an obvious easy solution.

Virtualenv is that solution. (To the point that it’s built-in to Python 3 as the “venv” module: python3 -m venv)

When installing html5lib from your requirements.txt, which points to a specific commit which is older than the system html5lib (1.0.1), pip decides that it would be a good idea to uninstall the system one, which obviously it cannot do.

Interestingly, html5lib is pinned by a third-party dependency not ourselves. This “one update” will result in a cascade of version bumps within our own requirements.txt, requiring some substantial testing before deployment. I’ll add this version bump to the backlog.

Back before Python 3 was a thing, virtualenv defaulted to including the global site-packages directory and users would typically invoke it with --ignore-site-packages to prevent exactly this type of conflict. (Today ignoring global site packages is the default.) It’s actually fairly impressive to me that you have a Python deployment architecture that doesn’t make use of virtual environments. Venv greatly improves isolation and reproducibility!

The update to the build process is simple:

virtualenv --python=python$PYTHON_VERSION venv_$APP_ID
source venv_$APP_ID/bin/activate
# proceed with app installation, “python” is now the chosen Python version via updated path.

The update to the run process is similarly simple: either ensure code is executed using the venv-contained bin/python executable (e.g. by sh-bang reference or explicit invocation), or the activate script is evaluated (and variables sourced into the current runtime) prior to launch. The on-disk structure is essentially unimportant; in development I utilize these environments in several structures:

1. Application source container. The virtual environment itself is where I check out the application code into, either as the venv itself, or as a contained src folder to maintain the LFS-like structure. (With third-party checkouts going in $VIRTUAL_ENV/usr/src via the --src argument to pip.) Git is told to ignore the venv-specific files.

2. A variation on the above places the virtual environment itself within the project folder as the .venv folder. Keeps things cleaner in Finder, keeps Git unaware.

3. Centralized environments. Utilizing the hidden .venv approach, it is often a symlink to one from my central collection of environments within ~/Library/Environments.

All of the above is automated and integrated into my Zsh shell setup such that entering a directory that is a venv or contains a .venv will automatically activate, deactivating when exiting the path. Zero friction use, no need for additional tools like pipenv or virtualenvwrapper.

— Alice.