How to: make Python dependencies installed via pip work on NixOS

nixos_python_patch_venv_bins.md

How to: make Python dependencies installed via pip work on NixOS

The issue

• You are using NixOS
• You start working on a Python project
• You manage the dependencies in a classic Python virtual environment using pip or poetry

You won't be able to import some of those libraries.

Example

$ nix run nixpkgs#python3 -- -m venv .venv
$ . .venv/bin/activate
(.venv) $ pip install numpy
(.venv) $ python -c 'import numpy'

You'll get for instance:

libz.so.1: cannot open shared object file: No such file or directory

What happens

When we peak at the installation of numpy, there are a bunch of compiled files (executables or libraries). Some of those are not properly linked.

$ nix shell nixpkgs#file --command find .venv/lib/python3.9/site-packages/numpy -type f -executable -exec sh -c "file -i '{}' | grep -qE 'x-(.*); charset=binary'" \; -print -exec sh -c "ldd '{}' | grep 'not found'" \;
.venv/lib/python3.9/site-packages/numpy/fft/_pocketfft_internal.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/bit_generator.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_common.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_pcg64.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_mt19937.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_bounded_integers.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_philox.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/mtrand.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_sfc64.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_generator.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/linalg/_umath_linalg.cpython-39-x86_64-linux-gnu.so
        libz.so.1 => not found
.venv/lib/python3.9/site-packages/numpy/linalg/lapack_lite.cpython-39-x86_64-linux-gnu.so
        libz.so.1 => not found
.venv/lib/python3.9/site-packages/numpy/core/_rational_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_operand_flag_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_umath_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_multiarray_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_struct_ufunc_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_simd.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so
        libz.so.1 => not found

In the example above, we can see that lapack_lite.cpython-39-x86_64-linux-gnu.so, one of the binary files provided by the numpy wheel, does not find the libz.so.1 library.

How to fix

1. Install the dependencies, or enter a shell

   $ nix shell nixpkgs#file nixpkgs#patchelf github:GuillaumeDesforges/nix-patchtools
   

2. Set the environment variable libs (used by autopatchelf) to a list of paths to all the required libraries.

   $ export libs=$(nix eval --expr 'let pkgs = import (builtins.getFlake "nixpkgs") {}; in pkgs.lib.strings.makeLibraryPath (with pkgs; [ gcc.cc zlib glibc ])' --impure | sed 's/^"\(.*\)"$/\1/'):$(find $(pwd) -name '*.libs' | tr '\n' ':' | sed 's/:$//')
   

3. Save the following script to a file and run it

   TMP="__tmp"
   mkdir -p $TMP
   cd $TMP
   
   find "$(realpath "../.venv")" -type f -executable -exec sh -c "file -i '{}' | grep -qE 'x-(.*); charset=binary'" \; -print \
     | while read file
       do
         echo "file=$file"
         RPATH=$(patchelf --print-rpath "$file")
         echo "rpath=$RPATH"
         autopatchelf "$file"
         if [[ "$RPATH" =~ .*(^|:)[^/].* ]]; then
           echo "repatch rpath"
           patchelf --set-rpath "" "$file"
           autopatchelf "$file"
         fi
       done
   
   cd ..
   rm -R $TMP

After running the script, check the result:

$ . .venv/bin/activate
(.venv) $ python -c 'import numpy'