Python package managers and environments

pythonenvs.md

uv

Experimental but very promising pip replacement that handles package managing as well as virtual environments and Python version management.

uv comes included with uvx, an alias for uv tool run. uvx allos you to install and execute command-line tools on an ephemeral environment.

Python versions

Note that you don't have to actively install a Python version! uv will automatically fetch the required Python version for your project.

1. Install a Python version (or many)
   • uv python install 3.11 3.12
2. View your installed Python versions
   • uv python list

Creating and managing projects

A project is a folder with some preconfigured files for ease of use, the most important being pyproject.toml, which defines the project's dependencies.

1. Create a project
   • uv init my_project
   • This will create a my_project folder in the current directory with the following files:
     • .python-version
     • README.md
     • hello.py
     • pyproject.toml
   • The project folder will NOT contain a virtual environment by default.
2. Add packages
   • uv add package
   • You need to execute this from the project's folder.
   • If no environment is present, uv will automatically create it.

Creating and managing environments

For when you want to create an environment on an already existing project folder.

1. Create an environment
   • uv venv > creates an environment at .venv.
   • uv venv name > creates an environment at name.
   • uv venv --python 3.11 > creates an environment with the specified Python version.
2. Activate the environment
   • source .venv/bin/activate > you may change .venv to whatever folder you used for your venv.
3. Deactivate the environment
   • deactivate
4. Install packages from your requirements file (with the venv activated)
   • uv pip install -r requirements.txt
   • uv pip install -r pyproject.toml
5. Add a single package to the venv (with the venv activated)
   • uv add package
6. Remove a package (with the venv activated)
   • uv remove package
7. If your requirements.txt (or pyproject.toml) file has changed, sync the environment to make sure that all packages match:
   • uv pip sync requirements.txt
8. Run code without activating the venv
   • uv run python my_app.py
9. Export dependencies to requirements.txt
   • uv export --format requirements-txt
   • Note that the exported file will be a lockfile with pinned versions.
   • The actual lockfile used by uv is uv.lock. Both pyproject.toml and uv.lock are more modern solutions that requirements.txt.

Cleanup

1. Remove an environment
   • Simply remove the .venv folder.
2. Clean the cache
   • uv cache clean
   • This will delete the entire cache!
3. Clean the cache of unused entries.
   • uv cache prune
   • Much safer than the previous command.

Conda

Manages packages and virtual environments for you. Good for Data Science but it sucks for dockerizing.

Creating and managing

1. Create a virtual environment
   • conda create --name my_env_name python=3.8 or whatever Python version you may need.
2. List available envs (2 different ways
   • conda env list
   • conda info --envs
3. Activate virtual env
   • conda activate my_env_name
4. Deactivate current environment
   • conda deactivate
5. If pip doesn't work with a fresh conda install:
   • conda install pip
6. Install project dependencies (listed in requirements.txt file)
   • conda install --file requirements.txt
   • pip install -r requirements.txt
7. Delete an old environment
   • conda remove --name my_env_name --all
   • conda env remove -n my_env_name
8. Update conda
   • conda update conda
9. Update all packages in the current environment
   • conda update --all
10. Update all packages in another env
    • conda update -n my_env_name --all
11. List installed packages in current environment
    • conda list
12. Add conda-forge channel
    • conda config --add channels conda-forge
13. Check conda channels
    • conda config --show channels
14. Remove conda-forge channel
    • conda config --remove channels conda-forge
15. Create an environment file from your current environment.
    • conda env export --from-history > environment.yml
16. Create a new environment and install dependencies listed in YML file.
    • conda env create -f environment.yml
17. If you don't want the base environment to load automatically whenever you open a new shell, change the configs:
    • conda config --set auto_activate_base false

Cleanup

1. List all available environments
   • conda env list
2. Remove a specific environment
   • conda env remove --name environment_name_to_be_deleted
3. Clean up the cache to remove packages and stuff
   • conda clean --all

Replacing Conda with something better

Take a look at Pixi as a drop-in replacement for Conda.

venv

venv is included with Python. It's clunkier than some other tools but it's available mostly everywhere. It allows you to create virtual environments tied to a specific project because they are contained within a subfolder inside your project's folder.

Keep in mind that virtual environments created with venv are not activated automatically.

Creating and managing

1. Create a virtual environment.
   • python -m venv path/to/virtual/environment/folder
   • By convention, the usual virtual environment folder is called .venv and placed in the root folder of your project. In these notes, it is assumed that all paths begin from the project's root folder and that .venv is the chosen name for the virtual environment.
   • The Python version of the venv will be the same you're using at the moment. I recommend using pyenv for managing your Python versions; you can activate a local Python version on the project folder and generate the venv afterwards.
2. Activate the virtual environment.
   • source .venv/bin/activate
3. Install dependencies from a requirements.txt file (you need to be on an activated venv).
   • pip install -r requirements.txt
   • You can use any other pip commands, such as pip install package_name and pip list to see installed packages.
4. Deactivate environment.
   • deactivate

Cleanup

1. Delete the venv folder to get rid of it.

• rm -rf .venv

Pipenv + Pyenv

pyenv is a Python version manager. It allows you to install and manage multiple Python versions.

pipenv is a Python virtualenv management tools. pipenv does not have built-in package search; make sure you search for the packages at PyPI.

Be aware that pipenv has some issues (very slow dependency lock, non-standard config file format) and that there are newer tools that may supplant it, such as PDM or uv.

Installation

1. pyenv
   • https://github.com/pyenv/pyenv
   • pyenv can be installed with either Brew or with the automatic installer script.
   • For Windows, there is pyenv-win but I have not tested it.
   • For Ubuntu 22.04 LTS, make sure you run the following before installing:
     • sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm xz-utils tk-dev libffi-dev liblzma-dev python3-openssl
2. pipenv
   • https://pipenv.pypa.io/en/latest/index.html
   • Install locally for your user with pip install pipenv --user

Usage

The environments are based on the folder you're on. There is no need to manually name them, and there is no environment activation to take care of per se.

1. (Optional) Install the Python version you need
   • pyenv install 3.11
2. Create a new virtual environment with pipenv and choose the Python vrsion you want.
   • pipenv install --python 3.11
   • pipenv should automatically use pyenv to install the specified version if it is not found on your system.
   • If a requirements.txt file is present, it will be used to install your packages. If a Pipfile is present, it will be used instead of requirements.txt because it has priority.
3. If you have a requirements.txt and a Pipfile and want to prioritize requirements.txt, you can choose which file to use in order to install packages:
   • pipenv install -r requirements.txt
4. Install a single package (this will modify Pipfile and Pipfile.lock, but WILL NOT modify requirements.txt if it exists):
   • pipenv install some_package
   • pipenv install some_package=1.0
5. If a Pipfile.lock file already exists, you can install the packages from it.
   • pipenv sync
6. Update packages
   • pipenv update > updates all packages
   • pipenv update <package> updates a single package and its sub-dependencies
7. Access the pipenv shell (necessary for enabling the virtualenv and for your script to find the installed packages).
   • pipenv shell
8. Exit a pipenv shell.
   • exit
9. Install all dependencies to the system, avoiding virtualenvs entirely (useful for deployment in containers)
   • pipenv install --system --deploy > Use this in your Dockerfile.

Cleanup

Pipenv environments are usually in ~/.local/share/virtualenvs/; inspect the directory to see all the virtualenvs.

To remove a virtualenv, navigate to its project directory and run:

pipenv --rm

After deleting the environment/s, you may want to clean up the pip cache:

pip cache purge

Installing pipenv on Docker Dev Environment

• Update apt

  sudo apt update && sudo apt upgrade
  

• Install pip

  sudo apt install python3-pip
  

• Install pyenv. Follow instructions here: https://github.com/pyenv/pyenv-installer

• Install pipenv. Follow instructions here: https://pipenv.pypa.io/en/latest/