python-packaging-1.md

I have had to understand Python packaging in depth on a number of occasions lately, both in my private development efforts and at my place of employment. For my own sake and simply to help others, I'd like to share what I consider valuable insight and information -- to save others time and effort, if mine was anything to judge by.

Python packaging has come further since people had to express everything with setup.py and more importantly without pyproject.toml. PEP 517 standardises how tools like Pip will execute building, in fact, and the standardisation basically makes it an entirely customizable process starting already with the so-called build backend. That's the value of build-backend in your pyproject.toml. A number of build backends have been developed now -- Poetry, Flit, Hatchling and others. Setuptools, a further development of disttools that used to ship with Python, has also been standardized to offer a build backend -- if the pyproject.toml specifies build-backend = "setuptools.build_meta", it means that the build_meta object (module or function) of setuptools module/package is going to execute deployment. This is PEP 517 stuff.

The good news is that if you for some reason cannot or would not use the backend to your advantage, you can write your own, including something called in-tree build backend -- one whose module(s) are provided as part of the project defined in the pyproject.toml. Together with backend-path you can make the in-tree backend do your building, in a manner that you control entirely.

Setuptools, for one, uses setup.py -- courtesy of its distutils legacy, in part. Which means that if you have a setup.py in the project and a pyproject.toml, in all likelihood setuptools.build_meta is the build backend in use, because said build backend is what will use setup.py, giving you a measure of control over your building process through use of Setuptools API.

You don't have to have a setup.py when using Setuptools, obviously -- in which case pyproject.toml metadata is what instructs Setuptools. It's just that you can't do everything with just the set of properties in the latter file that are standard and/or understood by Setuptools. What if you want to run an arbitrary program as part of your build process?

In one of the projects I have been developing, for convenience I have a set of Python modules in a "raw" (template) state -- they neither can nor are meant to be imported as-is, but instead are fed to a custom template processor that then generates the module content proper. I do that because in certain aspects Python is limited in terms of syntax, and if you, say, have 50 classes that you want to express for at least MyPy's (type checker) sake, I simply am not going to type 50 class definitions. Furthermore, MyPy doesn't "understand" dynamic type creation -- so creating 50 class definitions with a single for loop, for intance, is also out of the question. The way I solved it is with a template that also happened to be a Python file, which upon execution, produces, you guessed it, another Python file, this time containing the definitions in all their verbose, repetitive glory.

The problem then is how to include mechanisms like the above template processing when building the package for distribution? Exactly. Setuptools is perfectly capable of packaging your e.g. src directory into the so-called "sdist" (source distribution package, a tar.gz file, you probably recall seeing these being offered) and then transitively into the so-called "wheel" (the de-facto installable package that Pip will use for installing the package at the end-user's). But it won't invoke any extra steps through any magic written in pyproject.toml, not to my knowledge (if I am wrong, please feel free to correct me). In comes Setuptools API and setup.py that lets you use it.

Let's say your custom building step(s) -- like template processing -- is expressed with Make. Make is, after all, a popular tool for doing this kind of thing, owing to the body of C and C++ code out there, in the very least. So you have a Makefile that expresses one way or another that a Python module in src is compiled (generated) from a corresponding template file in templates. Here's more or less the minimum you'd need to do to have Setuptools invoke make when building the wheel, that is through pip install . or python -m build (the latter per PyPa recommendations):

import setuptools.command.build
from setuptools import Command, setup

import os
import os.path
import subprocess

class MakeCommand(Command):
    """Class of `setuptools`/`distutils` commands which invoke a `make` program.

    GNU Make (http://www.gnu.org/software/make) is currently assumed for providing `make`. The program is invoked in a manner where it changes the working directory to the build directory advertised for the command (utilizing `self.set_undefined_options` as hinted at by [the documentation](http://setuptools.pypa.io/en/latest/userguide/extension.html) which defers to `help(setuptools.command.build.SubCommand)`).

    The command is expected to produce build artefacts which will be added to the wheel.
    """
    build_lib: str | None
    def finalize_options(self) -> None:
        self.set_undefined_options('build_py', ('build_lib', 'build_lib'))
    def initialize_options(self) -> None:
        self.build_lib = None
    def run(self, *args, **kwargs) -> None:
        os.makedirs(self.build_lib, exist_ok=True)
        subprocess.check_call(('make', '-C', self.build_lib, '-f', os.path.realpath('Makefile')))

class BuildCommand(setuptools.command.build.build):
    sub_commands = [ ('build_make', None) ] + setuptools.command.build.build.sub_commands # Makes the `build_make` command a sub-command of the `build` command, which has the effect of the former being invoked when the latter is invoked (which is invoked in turn when the wheel must be built, through the `bdist_wheel` command)

setup(cmdclass={ 'build': BuildCommand, 'build_make': MakeCommand })

(the above is one of at least two different ways to accomlish the same thing, the other one using custom command classes much like the above but without use of so called sub-commands)

Too verbose? You can opt out of using Setuptools entirely, and offer your own, in-tree build backend. Say it's a single module that you put beside pyproject.toml, a enkindle.py (a synonym for "build" which isn't "build" for reasons I'll explain later):

import setuptools.build_meta
import subprocess
from sys import stderr
import os

def run_make():
    os.makedirs('src', exist_ok=True)
    subprocess.check_call(('make', '-C', 'src', '-f', os.path.realpath('Makefile')))

def build_sdist(*args, **kwargs):
    return setuptools.build_meta.build_sdist(*args, **kwargs)

def build_wheel(*args, **kwargs):
    run_make()
    return setuptools.build_meta.build_wheel(*args, **kwargs)

The above simply fulfills the contract described by PEP 517 and implemented by Python's build module (as identified at PyPi) and Pip.

The two aren't exactly the same -- there's nuances regarding which directory you'd want to run Make in so that build artefacts are generated where these will actually make it into the wheel being built, and so on. But in principle if Setuptools doesn't cut it you use a different backend, obviously -- or your own like above.

To be continued...