Allow Python on macOS M1 to import native libraries installed using Homebrew

Python native libs on macOS with M1.md

Your Problem

1. You are using Python on macOS M1 (or M2, etc) and are trying to install a Python library which uses a native library.
2. You install the Python library (e.g. using pip) and the native library (using Homebrew).
3. You get an error saying that the native library cannot be located even though it's installed using Homebrew.

Example: You are trying to use e.g. pyfluidsynth. You've installed pyfluidsynth with pip, and also ran brew install fluidsynth. However, import fluidsynth still gives an error:

ImportError: Couldn't find the FluidSynth library.

Explanation

What happens is that pyfluidsynth uses ctypes.find_library to locate the native fluidsynth library. However, find_library doesn't find libraries installed by Homebrew on M1 because Homebrew installs libraries in /opt/homebrew/lib (instead of /usr/local/bin used for Intel Macs) which is not in the default library search path.

Solutions

In order to solve this, you must tell Python to look in /opt/homebrew/lib. Use any of the following solutions depending on your setup. If you are using pyenv, jump straight to solution 3.

Solution 1 - Homebrew Python

Use Python installed by Homebrew (brew install python). Homebrew patches ctypes itself in their version of Python, so that native libraries installed using Homebrew are found by find_library.

This solution won't work if you want to use pyenv or any other installation of Python. It will only work with Homebrew-installed Python. You can apply the same patch to your own installation of Python, but if you use pyenv and multiple versions of Python this becomes quite annoying.

Solution 2 - DYLD_FALLBACK_LIBRARY_PATH

You can set an environment variable that will make ctypes.find_library look in Homebrew's library path:

export DYLD_FALLBACK_LIBRARY_PATH="/opt/homebrew/lib:$DYLD_FALLBACK_LIBRARY_PATH"

You can add this line to your shell profile (~/.zprofile) so that it happens in every shell. Restart your shell for the changes to take effect.

This solution still won't work if you use pyenv. Why? Because of a macOS security feature called SIP which prevents the variable DYLD_FALLBACK_LIBRARY_PATH from being forwarded to 'protected processes' such as the shell. And since pyenv uses a shell script to start Python, it won't get the new DYLD_FALLBACK_LIBRARY_PATH. You may be able to bypass this by partially disabling SIP in Recovery Mode.

Solution 3 - Usercustomize Script

Python executes a script called usercustomize every time it starts up. We can use that to make sure DYLD_FALLBACK_LIBRARY_PATH is set correctly in every Python process. We can even do that for all installed Python versions in a single place. And it works even with pyenv.

Create a usercustomize.py file: ~/.local/lib/global-packages/usercustomize.py. Paste in it the following script:

import os

homebrew_lib_dir = "/opt/homebrew/lib"
existing_lib_path = os.environ.get("DYLD_FALLBACK_LIBRARY_PATH", "")
os.environ["DYLD_FALLBACK_LIBRARY_PATH"] = homebrew_lib_dir + ":" + existing_lib_path

Then, make this usercustomize script be used by any Python interpreter (regardless of version): In your ~/.zprofile add the following line:

export PYTHONPATH="$HOME/.local/lib/global-packages:$PYTHONPATH"

To whoever is still reading this, export PYTHONPATH="$HOME/.local/lib/global-packages:$PYTHONPATH" is a security issue since it makes PYTHONPATH end with : if it's originally unset, which makes Python add your current working directory to sys.path.

This can lead to arbitrary code execution. For example, if you're in a directory that contains subprocess.py and you run any python script that imports subprocess, the subprocess.py file will be loaded from your current directory, and it can execute anything.

A better way to set it is something like

export PYTHONPATH="$HOME/.local/lib/global-packages${PYTHONPATH:+:$PYTHONPATH}"

...and for good measure, I assume the same should apply to setting DYLD_FALLBACK_LIBRARY_PATH, and to the usercustomize.py script. I changed it to

import os

homebrew_lib_dir = "/opt/homebrew/lib"
existing_lib_path = os.environ.get("DYLD_FALLBACK_LIBRARY_PATH", "")
if existing_lib_path:
    new_lib_path = homebrew_lib_dir + ":" + existing_lib_path
else:
    new_lib_path = homebrew_lib_dir
os.environ["DYLD_FALLBACK_LIBRARY_PATH"] = new_lib_path

See this StackOverflow link for more info.