Issue with urllib3 connectionpool under gevent. New HTTPS connection take an abnormally long time to instantiate.

gevent_urllib3_connectionpool_issue.py

from gevent.monkey import patch_all
patch_all()

import sys
import time

import gevent
import requests

import logging
logging.basicConfig(level=logging.INFO)


def task(url):
    tstart = time.time()
    client.get(url, allow_redirects=False)
    elapsed = time.time() - tstart
    print "  Task time %.2f ms" % (elapsed * 1000)


def test_gevent(url):
    print "Test flat gevent on %s" % url
    tstart = time.time()
    gevent.joinall([gevent.spawn(task, url) for x in range(10)])
    elapsed = time.time() - tstart
    print "Total Test Time = %.2f ms\n" % (elapsed * 1000)


def test_plain(url):
    print "Test plain on %s" % url
    tstart = time.time()
    [task(url) for x in range(10)]
    elapsed = time.time() - tstart
    print "Total Test Time = %.2f ms\n" % (elapsed * 1000)


if __name__ == '__main__':
    print "Python: %s\nRequests: %s\n\n" % (sys.version, requests.__version__)

    client = requests.Session()

    print "\n\nCONTROL\n"
    test_plain("http://www.google.com")
    test_plain("https://www.google.com")

    client = requests.Session()

    print "\n\nWith connection pool empty\n\n"
    test_gevent("http://www.google.com")
    test_gevent("https://www.google.com")

    print "\n\nWith connection pool full\n\n"
    test_gevent("http://www.google.com")
    test_gevent("https://www.google.com")

output

(gevent_tests)pior@saiga:~/dev/py/gevent_tests (master)$ python test.py 
Python: 2.7.9 (default, Apr  2 2015, 15:33:21) 
[GCC 4.9.2]
Requests: 2.7.0




CONTROL

Test plain on http://www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): www.google.com
  Task time 69.32 ms
  Task time 20.95 ms
  Task time 24.55 ms
  Task time 21.85 ms
  Task time 20.13 ms
  Task time 21.20 ms
  Task time 22.19 ms
  Task time 21.37 ms
  Task time 20.00 ms
  Task time 24.10 ms
Total Test Time = 266.63 ms

Test plain on https://www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.google.com
  Task time 119.81 ms
  Task time 22.94 ms
  Task time 21.55 ms
  Task time 22.86 ms
  Task time 20.20 ms
  Task time 21.66 ms
  Task time 22.26 ms
  Task time 30.73 ms
  Task time 22.55 ms
  Task time 20.40 ms
Total Test Time = 325.86 ms



With connection pool empty


Test flat gevent on http://www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (2): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (3): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (4): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (5): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (6): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (7): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (8): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (9): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (10): www.google.com
  Task time 70.03 ms
  Task time 54.14 ms
  Task time 60.55 ms
  Task time 50.64 ms
  Task time 61.01 ms
  Task time 55.96 ms
  Task time 80.57 ms
  Task time 74.38 ms
  Task time 72.10 ms
  Task time 87.69 ms
Total Test Time = 88.53 ms

Test flat gevent on https://www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (2): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (3): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (4): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (5): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (6): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (7): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (8): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (9): www.google.com
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (10): www.google.com
  Task time 379.21 ms
  Task time 362.82 ms
  Task time 361.10 ms
  Task time 358.79 ms
  Task time 370.21 ms
  Task time 377.56 ms
  Task time 379.76 ms
  Task time 382.06 ms
  Task time 390.75 ms
  Task time 395.21 ms
Total Test Time = 398.54 ms



With connection pool full


Test flat gevent on http://www.google.com
  Task time 42.80 ms
  Task time 38.93 ms
  Task time 32.53 ms
  Task time 36.01 ms
  Task time 47.12 ms
  Task time 45.90 ms
  Task time 41.39 ms
  Task time 43.49 ms
  Task time 34.34 ms
  Task time 37.67 ms
Total Test Time = 53.55 ms

Test flat gevent on https://www.google.com
  Task time 32.31 ms
  Task time 35.69 ms
  Task time 33.66 ms
  Task time 33.45 ms
  Task time 33.28 ms
  Task time 29.83 ms
  Task time 32.92 ms
  Task time 32.61 ms
  Task time 33.60 ms
  Task time 36.43 ms
Total Test Time = 45.07 ms

Did you ever figure this out?

Sorry, I have no recollection of the outcome 😬