Python requests SSLError on MacOS - resolution

.gitignore

.idea/
__pycache__/

README.md

Resolution to Python requests library SSLError on MacOS

I've come across this issue many times, and each time I have to search through layers of stackoverflow.com answers to find the one that is applicable to me, so now I am recording how I resolve this error in a gist so that maybe I will remember to just look at the gist next time.

TL;DR

Set environment variable

REQUESTS_CA_BUNDLE=/usr/local/etc/ca-certificates/cert.pem

when executing any program that uses the requests package.

Long explanation

Environment

The environment in which this occurs has these features:

• MacOS (Monterey currently, but it's happened prior to Monterey)
• Python 3.9 installed with Homebrew
• in a virtual environment, these are installed:
  • requests 2.27.1
  • certifi 2021.10.8

How the error manifests

I try to send an HTTP request to a domain such as drive.google.com that I am darn sure has a valid TLS certificate, and this exception is thrown:

>>> requests.get("https://drive.google.com/")
HTTPSConnectionPool(host='drive.google.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

Note that using Python's built-in HTTP facility, this does not happen:

>>> with urllib.request.urlopen('https://drive.google.com/') as response:
...   print(response.status, response.reason)
200 OK

So what's going on?

Some clues can be found by examining certificate paths as follows.

The certificates urllib trusts when it interacts with a server are:

>>> import ssl
>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile='/usr/local/etc/[email protected]/cert.pem', capath='/usr/local/etc/[email protected]/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/local/etc/[email protected]/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/local/etc/[email protected]/certs')

(Note that with the bundled-with-the-OS system Python interpreter /usr/bin/python3, urllib uses /etc/ssl/cert.pem.)

The certificates requests trusts when it interacts with a server are:

>>> import certifi
>>> certifi.where()
'$PATH_TO_VIRTUAL_ENVIRONMENT_DIR/lib/python3.9/site-packages/certifi/cacert.pem'

What we conclude from this is that for whatever reason the certificate trust store that is bundled with certifi is inadequate.

How do we resolve the issue?

The requests library will read the value of environment variable REQUESTS_CA_BUNDLE and interpret it as the pathname of a certificate trust store file.

So we can set this variable to a known-good trust store, e.g.

$ REQUESTS_CA_BUNDLE=/usr/local/etc/ca-certificates/cert.pem ./fetch.py

and execute without error.

Why not export the variable in .bash_profile

I hestitate to define the variable in .bash_profile because I'm not sure how frequently /usr/local/etc/ca-certificates will be updated, and I guess I'd rather trust the OS's default decisions about the best certificate trust store. This could be the wrong decision.

fetch.py

#!/usr/bin/env python3

"""Program that demonstrates an issue with the requests library TLS certificate verification on MacOS."""

import sys

import requests
from argparse import ArgumentParser

def main():
    parser = ArgumentParser()
    parser.add_argument("--domain", default="drive.google.com")
    args = parser.parse_args()
    url = f"https://{args.domain}/"
    try:
        with requests.get(url) as response:
            text = response.text
        print(text[:256], end="")
        if len(text) > 256:
            print()
            print(f"[clipped {len(text) - 256} characters]")
    except requests.exceptions.SSLError as e:
        print(f"GET {url} -> ", e, file=sys.stderr)
        return 1
    return 0

if __name__ == '__main__':
    exit(main())

requirements.txt

requests>=2.27.1