I'm a Python programmer and frequently work with the excellent virtualenv tool by Ian Bicking.
Virtualenv is a great tool on the whole but there is one glaring problem: the activate script that virtualenv provides as a convenience to enable its functionality requires you to source it with your shell to invoke it. The activate script sets some environment variables in your current environment and defines for you a deactivate shell function which will (attempt to) help you to undo those changes later.
This pattern is abhorrently wrong and un-unix-y. activate should instead do what ssh-agent does, and launch a sub-shell or sub-command with a modified environment.
The approach of modifying the user's current environment suffers from a number of problems:
- It breaks if you don't use a supported shell.
- A separate activate script must be maintained for each supported shell syntax.
- What do you do if you use no shell at all? (I.E. run programs in a virtualenv from a GUI.)
- If the
deactivatescript fails to un-set an environment variable, it may contaminate other environments. - If you want to edit
deactivateor any other function sourced into your environment, you have to kill your shell and re-source the script to see the changes take effect. - If you change the current directory from one to another virtual environment and forget to carefully
deactivateandactivateas you do so, you may end up using libraries from or making changes in the wrong one!
Virtualenv's activate suffers from a number of other warts as well:
-
You can't simply run the script; you have to learn and employ your shell's "source this script" builtin. Many non-experts frequently stumble over this distinction. Doing away with the recommendation to source a shell script should make virtualenv easier to use.
# This file must be used with "source bin/activate" *from bash* # you cannot run it directly -
In an attempt to preserve the user's old environment, it declares _OLD_VIRTUAL_PATH, _OLD_VIRTUAL_PYTHONHOME, and _OLD_VIRTUAL_PS1, and must define how to restore them upon deactivation. If you happen to want to modify
activateto override more variables specific to your environment, you have to do the same. -
Its default means to display whether or not a virtual environment is currently active (modifying the user's PS1 variable) is fragile. On Debian and Ubuntu boxes it becomes confusing if one enters a subshell, or uses a tool like
screenortmux. -
It is not executable, and not meant to be used as an executable, yet it lives in a a directory named
bin.
Entering and exiting a virtual environment should be like using ssh to connect to another machine. When you're done, a simple exit should restore you to your original, unmodified environment.
An example of a program that does this the Right Way is ssh-agent. In order to communicate the port that it uses to other programs, it must set some variables into the environment. It provides an option to do what virtualenv does, but the better way is to simply ask ssh-agent to launch your command for you, with a modified environment. ssh-agent $SHELL will launch a sub-shell for you with its environment already modified appropriately for ssh-agent. Most Debian and Ubuntu machines even launch X11 this way; see /etc/X11/Xsession.d/90x11-common_ssh-agent.
Another advantage to the subshell approach is that it is far simpler than the hoops virtualenv jumps through to activate and deactivate an environment. There's no need to set _OLD_ variables since the former environment is restored automatically. There's no need for a deactivate function.
Finally, employing a prompt context variable instead of messing with PS1 would allow the user to define how that information is presented.
To differentiate, I'm calling this approach "inve" as in "inside this virtual environment, ..." I'll happily take name suggestions.
How do we make an executable like ssh-agent that launches a subcommand with a modified environment? Easy. Call this my_launcher:
#!/bin/sh
export MY_VAR=xyz
exec "$@"
Calling "my_launcher firefox" will launch firefox with MY_VAR set to 'xyz' in its environment. The environment where "my_launcher" is called from will not be disturbed.
Let's now examine bin/activate to see what we can throw away if we assume that the system takes care of restoring the environment for us when we exit. We don't need the deactivate shell function at all. We don't need any _OLD_ variables. We don't mess with the prompt. What's left?
export VIRTUAL_ENV="/home/mike/var/virtualenvs/myvirtualenv"
export PATH="$VIRTUAL_ENV/bin:$PATH"
unset PYTHON_HOME
That's it. Three lines, down from 76. Down from 187 if you count all variants for other shells.
Wrap this with the launcher technique above, call it inve, and ./bin/inve $SHELL spawns a new subshell in the active virtualenv. What if you want a no-argument invocation to default to spawning an activated shell? This is the entire script:
#!/bin/sh
export VIRTUAL_ENV="/home/mike/var/virtualenvs/myvirtualenv"
export PATH="$VIRTUAL_ENV/bin:$PATH"
unset PYTHON_HOME
exec "${@:-$SHELL}"
Now bin/inve does what bin/activate should. By the way: this works for all shells. bash, zsh, csh, fish, ksh, and anything else, with one script.
Some users source bin/activate from within their own shell scripts, which I don't find quite as offensive.
ssh-agent also supports this style of use. It too has to deal with the syntax differences between shells to do so. It's not hard to enable this; here's one proposal.
#!/bin/sh
# As above, do what's needed to activate
export VIRTUAL_ENV="/home/mike/var/virtualenvs/myvirtualenv"
export PATH="$VIRTUAL_ENV/bin:$PATH"
unset PYTHON_HOME
# If the first argument is -s or -c, do what ssh-agent does
if [ "$1" = "-s" ]; then cat <<- DONE
export VIRTUAL_ENV="$VIRTUAL_ENV";
export PATH="$PATH";
unset PYTHON_HOME;
DONE
elif [ "$1" = "-c" ]; then cat <<- DONE
setenv VIRTUAL_ENV "$VIRTUAL_ENV";
setenv PATH "$PATH";
unset PYTHON_HOME;
DONE
# Otherwise, launch a shell or subcommand
else
exec "${@:-$SHELL}"
fi
Now inve supports the same -s and -c options that ssh-agent does. Where one might previously have written a script like this:
#!/bin/sh
source ./activate
... (commands) ...
One would now write instead:
#!/bin/sh
eval `./inve -s`
... (commands) ...
Or, for csh:
#!/bin/csh
eval `./inve -c`
... (commands) ...
Unfortunately, I don't know if this "eval the output of a command" technique works for all possible shells.
I find it convenient to employ a "system-level" inve script that lives in my system $PATH, that I can run from anywhere within any virtual environment, and without specifying the full path to 'ENV/bin/inve'. This goes against the intention that "virtualenvs are self-sufficient once created" so I'm not advocating this technique be used instead of ENV/bin/inve.
#!/bin/sh
# inve
#
# usage: inve [COMMAND [ARGS]]
#
# For use with Ian Bicking's virtualenv tool. Attempts to find the root of
# a virtual environment. Then, executes COMMAND with ARGS in the context of
# the activated environment. If no COMMAND is given, activate defaults to a
# subshell.
# First, locate the root of the current virtualenv
while [ "$PWD" != "/" ]; do
# Stop here if this the root of a virtualenv
if [ \
-x bin/python \
-a -e lib/python*/site.py \
-a -e include/python*/Python.h ]
then
break
fi
cd ..
done
if [ "$PWD" = "/" ]; then
echo "Could not activate: no virtual environment found." >&2
exit 1
fi
# Activate
export VIRTUAL_ENV="$PWD"
export PATH="$VIRTUAL_ENV/bin:$PATH"
unset PYTHON_HOME
exec "${@:-$SHELL}"
Until an inve-like script gets created in virtualenv bin/ directories, this system-level script will allow you to immediately use the subshell technique with all existing virtualenvs. If ever the inve script does land in virtualenv's bin/, this system level script could be simply a helper that searches for and invokes ENV/bin/inve:
# Locate the root of the current virtualenv
... (same as above) ...
# Activate
exec bin/inve "$@"
But what about the prompt? Build a PS1 that does the right thing everywhere without needing to be modified to suit a particular purpose. I tend to have a function that collects all the context info this way, in my .bashrc:
function ps1_context {
# For any of these bits of context that exist, display them and append
# a space.
virtualenv=`basename "$VIRTUAL_ENV"`
for v in "$debian_chroot" "$virtualenv" "$PS1_CONTEXT"; do
echo -n "${v:+$v }"
done
}
export PS1="$(ps1_context)"'\u@\h:\w\$ '
This lets the user control their PS1 and it works everywhere, no matter how many subshells or screen sessions you're nested into. This is the only piece that has to be customized per-shell.
While using activate is intended only a convenience and is not necessary to work within a virtual environment, most of programmers I know treat it as a black box and never do without it. I suspect that, in part, the complexity of the script is what prevents more programmers from avoiding it.
Perhaps the worst part about a popular, useful tool like virtualenv using this antipattern is that many other programmers are adopting it as normative and using it for their own work. virtualenvwrapper and dustinlacewell/capn are two examples. Stop doing this, everyone!
I wrote this back in March 2012, and cobbled together at that time some scripts that worked well-enough for me to get away from virtualenv's bad behavior. After some encouragement and feedback from the community I started a patch to the virtualenv project that would make it use this method, and added some adapter code that would avoid breaking the workflow for those used to the current-shell-modification behavior, but I still haven't finished it. Meanwhile, some great projects have sprung up and properly implemented this idea:
- berdario's invewrapper
- sashahart's vex
I'm super happy about this. I don't mind parallel implementations; it's all open source and we can build upon one another's good ideas.