notes on using libpython-clj with pyenv and pyenv-virtualenv

(copied from: clj-python/libpython-clj#2 (comment))

Relying on a version of python that is installed (here I don't mean using something like NixOS or Guix, but rather more a traditional arrangement) on one's system seems like asking for trouble -- e.g. the OS has its own priorities so who knows what will happen after an update, or perhaps one needs to test against different versions of python, etc.

pyenv + virtualenv seems like it might work for Linux/macos and IIUC there is a similar pyenv-win project so may be that's an option.

So along those lines...had some luck getting things to work with pyenv + virtualenv without miniconda.

Some digging turned up this text:

For instance, if the Python executable is found in /usr/local/bin/python, it will assume that the libraries are in /usr/local/lib/pythonX.Y. (In fact, this particular path is also the “fallback” location, used when no executable file named python is found along PATH.) The user can override this behavior by setting the environment variable PYTHONHOME, or insert additional directories in front of the standard path by setting PYTHONPATH.

via: https://docs.python.org/3/c-api/intro.html#embedding-python

According to the python man page:

  PYTHONHOME
         Change the location of the standard Python  libraries.   By  de‐
         fault,  the  libraries are searched in ${prefix}/lib/python<ver‐
         sion> and  ${exec_prefix}/lib/python<version>,  where  ${prefix}
         and  ${exec_prefix} are installation-dependent directories, both
         defaulting to /usr/local.  When $PYTHONHOME is set to  a  single
         directory, its value replaces both ${prefix} and ${exec_prefix}.
         To specify different values for these, set $PYTHONHOME to ${pre‐
         fix}:${exec_prefix}.

  PYTHONPATH
         Augments  the  default search path for module files.  The format
         is the same as the shell's $PATH: one or  more  directory  path‐
         names   separated   by  colons.   Non-existent  directories  are
         silently ignored.  The default search path is  installation  de‐
         pendent, but generally begins with ${prefix}/lib/python<version>
         (see PYTHONHOME above).  The default search path is  always  ap‐
         pended  to  $PYTHONPATH.  If a script argument is given, the di‐
         rectory containing the script is inserted in the path  in  front
         of  $PYTHONPATH.  The search path can be manipulated from within
         a Python program as the variable sys.path.

With this info in hand, the following worked here (Linux) assuming pyenv and pyenv-virtualenv (its plugin for virtualenv) are installed:

Use pyenv to install some version of python (e.g. 3.8.5):

pyenv install 3.8.5

This is used as a base for a variety of virtualenvs later. Installing a non-system-tied version of python here helps to reduce potential breakage from system-related changes.

Prepare a project directory:

mkdir my-project

Within the project directory, indicate which version of python to use for the project:

cd my-project && pyenv local 3.8.5

Create a virtualenv (presumably one can make others later if necessary) for the project:

pyenv virtualenv 3.8.5 my-project-virtualenv-1

Activate the virtualenv:

pyenv activate my-project-virtualenv-1

If one skips this step, dependencies installed (e.g. via pip in the next step), may end up in the base python directory. If one works with multiple projects, this is probably not desired.

Install any dependencies / modules / libraries, e.g. tree_sitter:

pip install tree_sitter

The "installed" piece should end up in the virtualenv, keeping the base python installation "fresh" and resuable for other projects or virtualenvs.

Appropriately set PYTHONHOME and PYTHONPATH:

PYTHONHOME=$HOME/.pyenv/versions/3.8.5
PYTHONPATH=$HOME/.pyenv/versions/my-project-virtualenv-1/lib/python3.8/site-packages

IIUC, setting PYTHONHOME is about ensuring that python knows where its standard libraries are. I think tweaking this may help address this issue: clj-python/libpython-clj#96 (comment).

IIUC, setting PYTHONPATH to point at a part within the virtualenv may help in ensuring that python can find additional libraries (e.g. those installed in step 6).

With these preparations in place, had some success in getting a project based on a slightly updated libpython-clj template to work, including interop access of a python library installed at step 6.

sogaiu/libpython-clj-with-pyenv-and-pyenv-virtualenv.md