This came out of debugging the error net/http: TLS handshake timeout
which seemed to only be happening for certain endpoints. The thing that these endpoints all had in common was that they used Let's Encrypt as their CA. Some googling lead me to think it might have something to do with OCSP, so I wrote a small application that could make an OCSP request for a specified certificate.
The issue ended up being that the server experience the TLS handshake timeout had port 80 blocked. Port 80 was required to make the OCSP HTTP request to check that the certificate wasn't revoked.
go run ocsp.go ./cert.cer