Skip to content

Instantly share code, notes, and snippets.

@liviaerxin

liviaerxin/python_setuptools_build_extensions.md

Last active March 8, 2023 06:49
Show Gist options
  • Save liviaerxin/f9678a01f2602541847d38fc1c1f36dd to your computer and use it in GitHub Desktop.
Save liviaerxin/f9678a01f2602541847d38fc1c1f36dd to your computer and use it in GitHub Desktop.
Download ZIP
Use setuptools to build extensions #python
Raw
python_setuptools_build_extensions.md

Why I Write This?

Some day, I was building plyvel in macOS(Apple Silicon). plyvel is the interface to LevelDB and build a Python Extension for using the leveldb C++ library. In that building process, I found:

❯ python setup.py build_ext -i -n
running build_ext
skipping 'src/plyvel/_plyvel.cpp' Cython extension (up-to-date)
building 'plyvel._plyvel' extension
creating build
creating build/temp.macosx-11-arm64-3.9
creating build/temp.macosx-11-arm64-3.9/src
creating build/temp.macosx-11-arm64-3.9/src/plyvel
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/opt/homebrew/include -I/opt/homebrew/opt/[email protected]/include -I/opt/homebrew/opt/sqlite/include -I/Users/siyaochen/Documents/python-packages/plyvel/.venv39/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/plyvel/_plyvel.cpp -o build/temp.macosx-11-arm64-3.9/src/plyvel/_plyvel.o -Wall -g -x c++ -std=c++11 -stdlib=libc++
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/opt/homebrew/include -I/opt/homebrew/opt/[email protected]/include -I/opt/homebrew/opt/sqlite/include -I/Users/siyaochen/Documents/python-packages/plyvel/.venv39/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/plyvel/comparator.cpp -o build/temp.macosx-11-arm64-3.9/src/plyvel/comparator.o -Wall -g -x c++ -std=c++11 -stdlib=libc++
creating build/lib.macosx-11-arm64-3.9
creating build/lib.macosx-11-arm64-3.9/plyvel
clang++ -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk build/temp.macosx-11-arm64-3.9/src/plyvel/_plyvel.o build/temp.macosx-11-arm64-3.9/src/plyvel/comparator.o -L/opt/homebrew/lib -L/opt/homebrew/opt/[email protected]/lib -L/opt/homebrew/opt/sqlite/lib -lleveldb -o build/lib.macosx-11-arm64-3.9/plyvel/_plyvel.cpython-39-darwin.so
...

The clang compiler use flags -I/opt/homebrew/include -I/opt/homebrew/opt/[email protected]/include -I/opt/homebrew/opt/sqlite/include in default. So if I have installed leveldb via brew install leveldb, the extension _plyvel.cpython-39-darwin.so would be compiled successfully.

However, when I switch to use the Offical Python, that failed! And I found the clang not use the above flags.

Important Differences In Different Python(s)

We often download the built Python from python official website and install it as the normal program setup procedure. However in macOS, we also always use Homebrew to install Python when we would like to skip the program setup process and uninstall it easly later.

Usually, we will not pay attention to the differences in the Homebrew's Python and the Official Python. But it may be very important for us to concern about them when we would like to build Python Extensions as the common building tools(such as setuptools) often follow the best advice to set for the compiler using the same c/c++ flags that the version of Python you are using was compiled with.

C/C++ Flags In Different Python(s)

Here are different flags from different Pythons using sysconfig,

Homebrew's Python:

❯ python -m sysconfig
Platform: "macosx-11-arm64"
Python version: "3.9"
Current installation scheme: "posix_prefix"

Paths: 
	data = "/Users/siyaochen/Documents/python-packages/plyvel/.venv39"
	include = "/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9"
	platinclude = "/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9"
	platlib = "/Users/siyaochen/Documents/python-packages/plyvel/.venv39/lib/python3.9/site-packages"
	platstdlib = "/Users/siyaochen/Documents/python-packages/plyvel/.venv39/lib/python3.9"
	purelib = "/Users/siyaochen/Documents/python-packages/plyvel/.venv39/lib/python3.9/site-packages"
	scripts = "/Users/siyaochen/Documents/python-packages/plyvel/.venv39/bin"
	stdlib = "/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/lib/python3.9"

Variables: 
	ABIFLAGS = ""
	AC_APPLE_UNIVERSAL_BUILD = "0"
	AIX_BUILDDATE = "0"
	AIX_GENUINE_CPLUSPLUS = "0"
	ALT_SOABI = "0"
	ANDROID_API_LEVEL = "0"
	AR = "/usr/bin/ar"
	ARFLAGS = "rcs"
	BASECFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic"
	BASECPPFLAGS = ""
	BASEMODLIBS = ""
	BINDIR = "/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/bin"
	BINLIBDEST = "/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/lib/python3.9"
	BLDLIBRARY = ""
	BLDSHARED = "clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk"
	BUILDEXE = ".exe"
	BUILDPYTHON = "python.exe"
	BUILD_GNU_TYPE = "arm-apple-darwin20.4.0"
	BYTESTR_DEPS = "\"
	CC = "clang"
	CCSHARED = ""
	CFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk"
	CFLAGSFORSHARED = ""
	CFLAGS_ALIASING = "-fno-strict-aliasing"
	CONFIGFILES = "configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in"
	CONFIGURE_CFLAGS = "-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk"
	CONFIGURE_CFLAGS_NODIST = "-I/opt/homebrew/include -flto -Wl,-export_dynamic -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden"
	CONFIGURE_CPPFLAGS = "-I/opt/homebrew/include"
	CONFIGURE_LDFLAGS = "-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk"
	CONFIGURE_LDFLAGS_NODIST = "-L/opt/homebrew/lib -flto -Wl,-export_dynamic -g"
	CONFIG_ARGS = "'--prefix=/opt/homebrew/opt/[email protected]' '--enable-ipv6' '--datarootdir=/opt/homebrew/opt/[email protected]/share' '--datadir=/opt/homebrew/opt/[email protected]/share' '--without-ensurepip' '--enable-loadable-sqlite-extensions' '--with-openssl=/opt/homebrew/opt/[email protected]' '--with-dbmliborder=gdbm:ndbm' '--enable-optimizations' '--with-lto' '--with-system-expat' '--with-system-ffi' '--with-system-libmpdec' '--enable-framework=/opt/homebrew/opt/[email protected]/Frameworks' '--with-dtrace' 'LLVM_AR=/usr/bin/ar' 'MACOSX_DEPLOYMENT_TARGET=11' 'CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk' 'CFLAGS_NODIST=-I/opt/homebrew/include' 'LDFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk' 'LDFLAGS_NODIST=-L/opt/homebrew/lib' 'CPPFLAGS=-I/opt/homebrew/include' 'CC=clang' 'PKG_CONFIG_PATH=/opt/homebrew/opt/[email protected]/lib/pkgconfig:/opt/homebrew/opt/readline/lib/pkgconfig:/opt/homebrew/opt/sqlite/lib/pkgconfig:/opt/homebrew/opt/xz/lib/pkgconfig' 'PKG_CONFIG_LIBDIR=/usr/lib/pkgconfig:/opt/homebrew/Library/Homebrew/os/mac/pkgconfig/11'"
	CONFINCLUDEDIR = "/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include"
	CONFINCLUDEPY = "/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9"
	COREPYTHONPATH = ""
	COVERAGE_INFO = "/private/tmp/pythonA3.9-20210503-46149-h530z/Python-3.9.5/coverage.info"
	COVERAGE_REPORT = "/private/tmp/pythonA3.9-20210503-46149-h530z/Python-3.9.5/lcov-report"
	COVERAGE_REPORT_OPTIONS = "--no-branch-coverage --title "CPython lcov report""
	CPPFLAGS = "-I. -I./Include -I/opt/homebrew/include"

Official Python:

❯ python -m sysconfig
Platform: "macosx-10.9-universal2"
Python version: "3.9"
Current installation scheme: "posix_prefix"

Paths: 
	data = "/Users/siyaochen/Documents/python-packages/plyvel/venv39"
	include = "/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9"
	platinclude = "/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9"
	platlib = "/Users/siyaochen/Documents/python-packages/plyvel/venv39/lib/python3.9/site-packages"
	platstdlib = "/Users/siyaochen/Documents/python-packages/plyvel/venv39/lib/python3.9"
	purelib = "/Users/siyaochen/Documents/python-packages/plyvel/venv39/lib/python3.9/site-packages"
	scripts = "/Users/siyaochen/Documents/python-packages/plyvel/venv39/bin"
	stdlib = "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9"

Variables: 
	ABIFLAGS = ""
	AC_APPLE_UNIVERSAL_BUILD = "0"
	AIX_BUILDDATE = "0"
	AIX_GENUINE_CPLUSPLUS = "0"
	ALT_SOABI = "0"
	ANDROID_API_LEVEL = "0"
	AR = "ar"
	ARFLAGS = "rcs"
	BASECFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic"
	BASECPPFLAGS = "-IObjects -IInclude -IPython"
	BASEMODLIBS = ""
	BINDIR = "/Library/Frameworks/Python.framework/Versions/3.9/bin"
	BINLIBDEST = "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9"
	BLDLIBRARY = ""
	BLDSHARED = "gcc -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -g"
	BUILDEXE = ".exe"
	BUILDPYTHON = "python.exe"
	BUILD_GNU_TYPE = "x86_64-apple-darwin20.4.0"
	BYTESTR_DEPS = "\"
	CC = "gcc"
	CCSHARED = ""
	CFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g"
	CFLAGSFORSHARED = ""
	CFLAGS_ALIASING = "-fno-strict-aliasing"
	CFLAGS_NODIST = ""
	CONFIGFILES = "configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in"
	CONFIGURE_CFLAGS = "-arch arm64 -arch x86_64 -g"
	CONFIGURE_CFLAGS_NODIST = "-std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden"
	CONFIGURE_CPPFLAGS = ""
	CONFIGURE_LDFLAGS = "-arch arm64 -arch x86_64 -g"
	CONFIGURE_LDFLAGS_NODIST = ""
	CONFIG_ARGS = "'-C' '--enable-framework' '--enable-universalsdk=/' '--with-universal-archs=universal2' '--with-computed-gotos' '--without-ensurepip' '--with-openssl=/tmp/_py/libraries/usr/local' '--with-tcltk-includes=-I/tmp/_py/libraries/usr/local/include' '--with-tcltk-libs=-ltcl8.6 -ltk8.6' 'LDFLAGS=-g' 'CFLAGS=-g' 'CC=gcc'"
	CONFINCLUDEDIR = "/Library/Frameworks/Python.framework/Versions/3.9/include"
	CONFINCLUDEPY = "/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9"
	COREPYTHONPATH = ""
	COVERAGE_INFO = "/private/tmp/_py/_bld/python/coverage.info"
	COVERAGE_REPORT = "/private/tmp/_py/_bld/python/lcov-report"
	COVERAGE_REPORT_OPTIONS = "--no-branch-coverage --title "CPython lcov report""
	CPPFLAGS = "-IObjects -IInclude -IPython -I. -I/Users/sysadmin/build/v3.9.5/Include"
	CXX = "g++"

Homebrew's Python, clang will use cflags such as -I/opt/homebrew/include, -I/opt/homebrew/opt/[email protected]/include, -L/opt/homebrew/lib, ...etc. In contrast, the Official Python will not introduce them in compilation.

References

Setting Compiler Flags

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment