Notes on compiling Python + Rust (PyO3) for wasm32-wasip1

Critical metadata

Date written: 2024-10-30
Python version: 3.12.7
Rust version: 1.82.0
PyO3 version: 0.22.5
zlib version: 1.3.1

Information referenced

Actual notes

WASI

First of all, I am not using wasi-sdk but instead only Homebrew LLVM and the wasi-sdk sysroot. This causes problems later.

Extract the wasi-sdk sysroot somewhere on the system. I extracted a second copy of the sysroot, for hacky reasons which will be explained later.

zlib

You need to obtain a copy of zlib, or else you will end up with a build of Python that cannot do anything, because it cannot load its runtime libraries, because the runtime libraries are compressed in a zip file, which cannot be decompressed due to lack of zlib.

CHOST=wasm32 CC="/opt/homebrew/opt/llvm/bin/clang --target=wasm32-wasi --sysroot=/Users/user/code/wasi-sysroot" AR="/opt/homebrew/opt/llvm/bin/llvm-ar" RANLIB="/opt/homebrew/opt/llvm/bin/llvm-ranlib" ./configure --static --prefix=/
make
make DESTDIR=/Users/user/code/wasi-sysroot install

If you don't pass CHOST then it tries to use libtool and then breaks for some reason.

The above compiles one version of zlib and then puts it into toplevel directories not separated by threading model. I don't know how to fix it. This is the reason for making a copy of the sysroot.

Python

Extract the Python source code. Make the following changes to Tools/wasm/wasi-env:

Change the WASI_SYSROOT="${WASI_SDK_PATH}/share/wasi-sysroot" line to WASI_SYSROOT="/Users/user/code/wasi-sysroot" (because it cannot be overridden from the environment)
Add --target=wasm32-wasi to each of the CC="${WASI_SDK_PATH}/bin/clang" CPP="${WASI_SDK_PATH}/bin/clang-cpp" CXX="${WASI_SDK_PATH}/bin/clang++" lines. This is because Homebrew LLVM doesn't default to building for WASM.

Invoke python3 Tools/wasm/wasm_build.py wasi. This should automagically compile Python as an executable and a static library.

Run the following:

cd builddir/wasi
make wasm_stdlib

This ends up building usr/local/lib/python312.zip and other related standard library files.

At this point, you can test using wasmtime run --dir .::/ python.wasm

PyO3

PyO3 can be added to a new binary crate in mostly the standard way, except auto-initialize is not allowed:

[dependencies]
pyo3 = { version = "0.22.5" }

use pyo3::{append_to_inittab, prelude::*};

#[pyfunction]
fn rustfunc(x: usize) {
    println!("Back in Rust! {x}")
}

#[pymodule(name = "testmodule")]
fn my_extension(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(rustfunc, m)?)
}

fn main() -> PyResult<()> {
    // Must do this manually
    append_to_inittab!(my_extension);
    // Must do this manually
    pyo3::prepare_freethreaded_python();
    Python::with_gil(|py| {
        let sys = py.import_bound("sys")?;
        let version: String = sys.getattr("version")?.extract()?;

        println!("Hello world, I'm Python {}", version);

        PyModule::from_code_bound(py, r"#
import testmodule
print(testmodule)
testmodule.rustfunc(12345)
        #", "main.py", "main").unwrap();
        Ok(())
    })
}

In order to make things actually compile, you need to build as follows:

RUSTFLAGS="\
    -L /Users/user/code/Python-3.12.7/builddir/wasi \
    -L /Users/user/code/wasi-sysroot/lib/wasm32-wasi \
    -C link-arg=-l:libwasi-emulated-signal.a \
    -C link-arg=-l:libwasi-emulated-getpid.a \
    -C link-arg=-l:libdl.a \
    -L /Users/user/code/Python-3.12.7/builddir/wasi/Modules/_decimal/libmpdec \
    -C link-arg=-l:libmpdec.a \
    -L /Users/user/code/wasi-sysroot/lib \
    -C link-arg=-l:libz.a \
    -L /Users/user/code/Python-3.12.7/builddir/wasi/Modules/_hacl \
    -C link-arg=-l:libHacl_Hash_SHA2.a \
    -L /Users/user/code/Python-3.12.7/builddir/wasi/Modules/expat \
    -C link-arg=-l:libexpat.a \
    -C link-arg=-l:libwasi-emulated-process-clocks.a" \
cargo build --target=wasm32-wasip1

You need these link arguments in order to statically link all of the relevant Python runtime bits as well as libwasi-emulated-*.

I don't currently know how to automatically determine what needs to be linked.

This quite possibly causes a wasi-sdk vs Rust mismatch unless you fix -Clink-self-contained. I have not yet dealt with this problem.

You can finally test by running wasmtime run --dir /Users/user/code/Python-3.12.7/builddir/wasi::/ target/wasm32-wasip1/debug/test_pyo3_wasi.wasm

TODO

Clean this up, automate it, and build many more libraries such as bz2, lzma, uuid, etc.

ArcaneNibble/notes_pyo3_wasm_wasi.md