matthewfeickert/README.md

Creating a Jupyter kernel for Python virtual environment on UChicago Analysis Facility

Reproduction of core content of ATLAS Talk Topic: Preferred way of creating a Jupyter kernel for Python virtual environment (ATLAS Internal, sorry)
Corresponding YouTube explainer video: Creating an IPython kernel for your venv on UChicago AF Jupyter Lab (no captions, sorry)

For robustness and reproduciblity, I would like to use a Python virtual environment when working with ml_platform:latest containers (easy enough) but I would also like to have IPython kernels that I make in that virtual environment be recognized by Jupyter Lab's launcher. I believe that the easiest way to do this is to just use the --user flag when creating an IPython kernel, but I wanted to know if there is an alternative / create this as an example Issue for others to reference.

Here's an example of what I'm doing in a ml_platform:latest container:

$ mkdir example
$ cd example
$ python3 --version
Python 3.8.10
$ python3 -m venv example-venv && . example-venv/bin/activate
$ python -m pip --quiet install --upgrade pip setuptools wheel
$ python -m pip list
Package       Version
------------- -------
pip           23.0
pkg_resources 0.0.0
setuptools    67.1.0
wheel         0.38.4
$ python -m pip --quiet install tensorflow ipykernel
$ python -m ipykernel install --help
usage: ipython-kernel-install [-h] [--user] [--name NAME] [--display-name DISPLAY_NAME] [--profile PROFILE] [--prefix PREFIX] [--sys-prefix]
                              [--env ENV VALUE]

Install the IPython kernel spec.

optional arguments:
  -h, --help            show this help message and exit
  --user                Install for the current user instead of system-wide
  --name NAME           Specify a name for the kernelspec. This is needed to have multiple IPython kernels at the same time.
  --display-name DISPLAY_NAME
                        Specify the display name for the kernelspec. This is helpful when you have multiple IPython kernels.
  --profile PROFILE     Specify an IPython profile to load. This can be used to create custom versions of the kernel.
  --prefix PREFIX       Specify an install prefix for the kernelspec. This is needed to install into a non-default location, such as a conda/virtual-env.
  --sys-prefix          Install to Python's sys.prefix. Shorthand for --prefix='/home/feickert/example/example-venv'. For use in conda/virtual-envs.
  --env ENV VALUE       Set environment variables for the kernel.
$ python -m ipykernel install --user --name="example-venv" --display-name="Example venv kernel"
Installed kernelspec example-venv in /home/feickert/.local/share/jupyter/kernels/example-venv

This then allows for viewing and using the kernel from Jupyter Lab's launcher. In the below screenshots you can see this as well as verify that the virtual environment that the kernel was created in is getting picked up and used. 👍