The user experience of Python on a minimal Debian or Ubuntu installation is bad. Core features like virtual environments, pip bootstrapping, and the ssl module are either missing or do not work like designed and documented. Some Python core developers including me are worried and consider Debian/Ubuntu's packaging harmful for Python's reputation and branding. Users don't get what they expect.
The problems can be easily reproduced with official Debian and Ubuntu containers in Docker or Podman. Debian Stable (Debian 10 Buster) comes with Python 3.7.3. Ubuntu Focal (20.04 LTS) has Python 3.8.5.
$ docker run -ti debian:stable
$ docker run -ti ubuntu:focal
# apt update
# apt install python3
venv is another Python standard library module. It provides support for creating lightweight "virtual environments". The venv module is available but dysfunctional. It cannot create virtual environments out of the box.
# python3 -m venv /tmp/venv
The virtual environment was not created successfully because ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
apt-get install python3-venv
You may need to use sudo with that command. After installing the python3-venv
package, recreate your virtual environment.
Failing command: ['/tmp/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']
Update Julien Palard wrote that one of his students ran into another issue with venv. Debian's venv can give an invalid advise when a user has multiple Python versions installed.
The ensurepip package is part of Python's standard library and provides support for bootstrapping the pip installer into an existing Python installation or virtual environment. The ensurepip package is missing on Debian/Ubuntu.
# python3 -m ensurepip
/usr/bin/python3: No module named ensurepip
# pip
bash: pip: command not found
After installation of python3-venv, the ensurepip package is failing with a different error message:
# python3 -m ensurepip
ensurepip is disabled in Debian/Ubuntu for the system python.
Python modules for the system python are usually handled by dpkg and apt-get.
apt-get install python-<module name>
Install the python-pip package to use pip itself. Using pip together
with the system python might have unexpected results for any system installed
module, so use it on your own risk, or make sure to only use it in virtual
environments.
# echo $?
1
The distutils package is mostly missing. Only the package root and distutils.version is available. The remaining code has been moved to python3-distutils by Debian/Ubuntu packagers. The python3-distutils is not installed with python3 and must be installed separately.
# python3
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils
>>> from distutils import sysconfig
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'sysconfig' from 'distutils' (/usr/lib/python3.7/distutils/__init__.py)
A minimal installation has no CA certificates because neither the python3 package nor OpenSSL libraries depend on ca-certificates.
- Debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960869
- Ubuntu bug: https://bugs.launchpad.net/ubuntu/+source/python3.6/+bug/1879310
>>> import urllib.request
>>> urllib.request.urlopen("https://pypi.org/")
Traceback (most recent call last):
...
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/lib/python3.7/urllib/request.py", line 1367, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/usr/lib/python3.7/urllib/request.py", line 1326, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>
Debian/Ubuntu have applied downstream patches to OpenSSL. The patches have caused breakage of user applications or Python's CI tests. Examples for issues and workarounds:
- https://bugs.python.org/issue41561
- https://bugs.python.org/issue31453
- https://bugs.python.org/issue26867
- https://github.com/python/cpython/blob/v3.6.12/Lib/test/test_ssl.py#L133-L147
- https://github.com/python/cpython/blob/v3.6.12/Lib/test/test_ssl.py#L149-L178
- https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1899878
- https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1917625
The lib2to3 package is moved to python3-lib2to3 package, which is not installed by default.
The tkinter package is not part of the default distribution. For once this is a good decision. tkinter depends on libtk and whole lot of X11 libraries. Graphical user interface libraries should not be installed by default on headless servers and containers. I just find it confusing that the tkinter package is provided by a python3-tk package and not by python3-tkinter.
Paul Ganssle added a zoneinfo implementation with timezons to Python 3.9, see PEP 615. The feature requires tzdata database. As of 2020-11-13 Debian and Ubuntu's python3.9 package are missing a dependency on the tzdata package. The zoneinfo module does not work without tzdata:
>>> import zoneinfo
>>> zoneinfo.available_timezones()
set()
>>> zoneinfo.ZoneInfo("CET")
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.9/zoneinfo/_common.py", line 24, in load_tzdata
raise ZoneInfoNotFoundError(f"No time zone found with key {key}")
zoneinfo._common.ZoneInfoNotFoundError: 'No time zone found with key CET'
NOTE The issue has been fixed by Anthony Sottile in Deadsnakes PPA, see comment.
UPDATE My launchpad bug 1904271 was closed as Invalid. Matthias wrote that tzdata is a required package and pointed to Debian policy. However the package is not installed by default in the official Debian and Ubuntu container images.
Virtualenvs contain de-vendored dependencies of pip and setuptools, https://bugs.launchpad.net/ubuntu/+source/python-virtualenv/+bug/1904945
Minimalization of Python installation is a legitimate effort. However a minimal installation of Python with core features missing should not be called a Python installation. Users should expect that package-manager install python3 gets them a working Python interpreter with majority of stdlib packages (with exception to tkinter GUI and test package).
I propose
- Debian's current minimized Python package
python3should rather be calledpython3-minimalor something similar. This package would still users to get a stripped down interpreter if they explicitly ask for it. apt install python3should provide a Python installation with working venv, ensurepip (*), distutils, and ssl modules.
(*) I define working ensurepip as python3 -m ensurepip does not fail and python3 -m pip works afterwards. It does not imply that stdlib's pip bundle must be shipped with Python distribution package. Debian could also provide an API compatible ensurepip facade and make python3 package depend on python3-pip.
Hi, I’m Thomas Wouters, and I’m one of the people on the Python Steering Council. (For the record, I’m also on the PSF Board of Directors, but I’m responding here solely with my Steering Council hat on. We did not discuss this response with the PSF Board, although they are aware of this discussion.)
We’ve been discussing this within the Steering Council for a while, and I think at this point our position is that this is a real, disruptive, common problem, but that it’s a Debian problem, not so much a Python one -- and that we would like the Debian project to acknowledge this and to continue to try and improve the situation.
I’m going to focus on the Python/virtualenv/pip situation here, since I can’t say much about the OpenSSL issues. I want to stress the fact that this is a real, wide-spread problem with a lot of user impact. It affects both new Python programmers and experienced ones. I see this in the #python IRC channel on Freenode on a daily basis, and often after people have tried to work around the problems in ways that mess up their system. While the correct way to deal with this may seem obvious to the Debian packagers, it’s decidedly not clear to end users, and they often end up following outdated or incorrect advice that leaves their Debian installations in an undesirable state. This then leads to seemingly innocuous changes breaking Python or something written in Python. Other steering council members have had similar experiences where people are confused as to why instructions on how to get started with Python do not work on Debian-based distributions.
I think we all understand that packaging something like Python is difficult, especially since it comes with a third-party package management tool, pip. Nevertheless users have a real need for pip, and often without understanding the difference between system installations, virtualenv, and --user installs. In the Python ecosystem it’s simply too common to install PyPI packages, or install directly from GitHub repos, or install different versions in different places. The Debian package management system can’t satisfy those needs (and I don’t believe it was ever intended to).
And I think it’s important to realise here that we’re not just talking about users who choose to use Debian. Debian is the basis for many other distros, for Docker images and other containers, for Raspberry Pi OS, etc, etc. This impacts many different types of users.
At the same time, I’m very hesitant to dictate solutions here -- even if it’s in the form of an informational PEP that duplicates what’s already in the Makefile -- because the Python core developers, as a group, are not the right people to do this. I say this for four reasons, really:
The core developers are not representative of the end user. We don’t really know how and why they use Python; whether they’re writing code in Python or just running third-party code; how the code is packaged up, how the Debian system they’re using is maintained, etc. We simply don’t know what Debian users care about in terms of their Python installation. We just see a lot of complaints that originate from the same root causes.
The core developers are by and large not Debian packagers, and don’t share their considerations. There can certainly be good reasons for Debian to split packages the way it does, or to not install pip. Mixing package managers that aren’t aware of each other is one of the problems people run into with Debian right now, and we don’t want to exacerbate that problem by dictating ‘python -m pip’ should always work.
The core developers aren’t the driving force behind the tools that are central to this problem -- virtualenv, pip and setuptools -- nor the best practices surrounding them. Virtualenv was a third-party development that was replicated in the standard library; pip, while bundled with Python, remains a separate project; setuptools is a fork of distutils that tried out yet another package format that’s since been deprecated (eggs). Best practices and extended tooling around this all (tox, nox, pipenv, poetry, etc) are all community efforts. An informational PEP dictating what should work might solve the current problem, but it’d always be lagging behind actual community practices.
Finally, and this may sound a little harsh, fundamentally the problem here is one between Debian and its users. It’s a problem that’s been dragging along for many years now, going at least as far back as when distutils was split off from the regular Python install. I don’t know if the answer is to change the default installation or to educate users better. The choice to apply the Debian packaging policy differently to provide a better user experience, or to make it more obvious what end users should be doing, or whatever other option there is to mitigate the end users’ problems, is one Debian packagers and policy setters need to make, not something the core developers should be dictating.
The main question now is whether the Debian project is aware of the problems, and whether there’s agreement on the seriousness of the issues. If the case is really that the best solution for end users requires Python, as a project, to dictate that solution -- well, so be it, but I would hope Debian already wants to do what’s best for users. Even though we don’t think we’re the best people to solve these problems, we are happy to provide guidance if desired.