dmtucker/1.17

dmtucker · 2018-03-04T01:19:38Z

1.17 Discussions

March 4, `#python` on Freenode

15:03    dstufft| ofek: what's up?
15:05       ofek| dstufft we were wondering where 1.17 and None come from https://github.com/ofek/pypinfo#downloads-for-a-project-by-python-version
15:06    dstufft| None just means no data was reported, could be old versions of pip, could be mirroring infrastructure, whatever
15:07    dstufft| 1.17 I have no idea tbh. My guess is it's some alternate implementation of Python that had it's own versioning, and instead of doing urllib-python/{python_version} it did 
                  urllib-python/{implementation_version} for the urllib host header
15:08       ofek| dtux runciter nedbat ^
15:08       ofek| dstufft thank you!
15:08     nedbat| ¯\_(ツ)_/¯
15:09    dstufft| ofek: basically all that data comes from the user-agents, which comes from introspecting the runtime, so our data is only as good as what the runtime provides (and in the case of older pip's 
                  and non-pip agents, sometimes with a healthy dose of assumptions at play too)
15:12       ofek| dstufft, ah okay, makes sense. I'm saving this convo for when this gets asked again :)

March 5, `#python` on Freenode

[14:53:16] <disi> dstufft: is there any way to get more info on the user agents passing python 1.17? i've been doing some digging without much luck yet. doesnt look like anything recognizable is being provided, but maybe the raw user-agents are available?
[14:54:22] <dstufft> disi: I'd have to dig it out of the archival logs
[14:54:46] <dstufft> we purposely throw that data away from BigQuery to avoid leaking private data people might have in their user agent that we didn't expect
[14:57:22] <dstufft> disi: I did this one before, I think it was literally just the urllib2 UA with "1.17" where "2.7" or "3.6" normally would go

dmtucker · 2018-03-04T23:30:59Z

What is 1.17 used for?

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 365 --limit 50 --where 'details.python = "1.17"' '' project
Served from cache: False
Data processed: 138.32 GiB
Data billed: 138.32 GiB
Estimated cost: $0.68

| project              | download_count |
| -------------------- | -------------- |
| kudu-python          |         28,210 |
| kazoo                |         26,628 |
| pytest               |         24,272 |
| pytest-xdist         |         24,132 |
| py                   |         23,960 |
| cryptography         |         23,954 |
| cffi                 |         23,803 |
| impyla               |         23,698 |
| setuptools-scm       |         23,033 |
| pycparser            |         19,441 |
| scandir              |         19,173 |
| azure-datalake-store |         19,136 |
| virtualenv           |         16,072 |
| requests             |         11,126 |
| jinja2               |         10,509 |
| botocore             |          9,708 |
| paramiko             |          9,315 |
| fabric               |          8,901 |
| sh                   |          8,796 |
| ordereddict          |          8,696 |
| ecdsa                |          8,686 |
| pycrypto             |          8,658 |
| flask                |          8,639 |
| cython               |          8,620 |
| six                  |          8,543 |
| simplejson           |          8,498 |
| docutils             |          8,421 |
| boto3                |          8,398 |
| jmespath             |          8,266 |
| futures              |          8,260 |
| numpy                |          8,133 |
| pyparsing            |          8,112 |
| python-dateutil      |          8,045 |
| psutil               |          8,005 |
| markupsafe           |          7,943 |
| werkzeug             |          7,925 |
| itsdangerous         |          7,903 |
| pbr                  |          7,882 |
| ipython              |          7,881 |
| pytest-random        |          7,869 |
| pexpect              |          7,836 |
| argparse             |          7,742 |
| thrift               |          7,741 |
| sqlparse             |          7,734 |
| readline             |          7,719 |
| pg8000               |          7,713 |
| docopt               |          7,704 |
| hdfs                 |          7,701 |
| allpairs             |          7,698 |
| cm-api               |          7,697 |

kudu-python, kazoo, and impyla are all Hadoop-related (2 are made by Cloudera).
- In 2016, Anaconda for Cloudera happened.
  - conda (Anaconda's pip) didn't add a proper user-agent until 4.1.0.
    - Before then, it relied on user-agent headers from requests which is built on urllib3.
      - requests vendors urllib3 and they skipped 1.17.
      - By default, urllib3 does not set a user agent.

dmtucker · 2018-03-05T01:13:45Z

How long has 1.17 been showing up?

dtucker@dtucker-wkstn:~ $ pypinfo --days 1095 --limit 36 --order download_month --where 'details.python = "1.17"' '' month
Served from cache: False
Data processed: 186.07 GiB
Data billed: 186.07 GiB
Estimated cost: $0.91

| download_month | download_count |
| -------------- | -------------- |
| 2018-03        |         16,157 |
| 2018-02        |         67,154 |
| 2018-01        |      1,898,377 |
| 2017-12        |         70,865 |
| 2017-11        |         64,577 |
| 2017-10        |         66,968 |
| 2017-09        |         63,809 |
| 2017-08        |         93,022 |
| 2017-07        |        628,639 |
| 2017-06        |        200,335 |
| 2017-05        |        148,618 |
| 2017-04        |         50,842 |
| 2017-03        |         53,755 |
| 2017-02        |         47,414 |
| 2017-01        |         50,651 |
| 2016-12        |         53,138 |
| 2016-11        |        129,537 |
| 2016-10        |        477,819 |
| 2016-09        |      1,013,797 |
| 2016-08        |        116,291 |
| 2016-07        |         81,521 |
| 2016-06        |         59,695 |
| 2016-05        |        151,872 |
| 2016-03        |          4,618 |
| 2016-02        |        226,267 |
| 2016-01        |        184,133 |

This doesn't seem to track with any particular Python version: https://gist.github.com/dmtucker/4e07eb51246f999546dec0dde1cb69f2
September 2016
- pypi.org became the default upload platform in September 2016.
January 2018
- January 2018 was the first PyCascades Event.
- Python 3.7.0b1, 3.5.5rc1 and 3.4.8rc1 all dropped.

dmtucker · 2018-03-05T23:17:46Z

Where does 1.17 come from?

Standard urllib seems like the most likely source (which is present in at least one fork):

$ python -c 'import sys, urllib, urllib2;  print(sys.version[:3], urllib.__version__, urllib2.__version__)'; python3 -c 'import urllib.request; print(urllib.request.__version__)'
('2.7', '1.17', '2.7')
3.5

PyPI associates urllib with the Python version.
That would make this issue specific to Python 2.
- The top downloads all seem to support it still.
- urllib3 is also a possibility.
  - By default, urllib3 does not set a user agent.

What agents may be using urllib?

bandersnatch?
conda?
easy_install switched to urllib2 in 0.4a2.
pip?
requests vendors urllib3 and skipped 1.17.

dmtucker · 2018-03-06T23:34:53Z

Where is 1.17 being used from?

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 365 --where 'details.python = "1.17"' '' country
Served from cache: False
Data processed: 81.80 GiB
Data billed: 81.80 GiB
Estimated cost: $0.40

| country | download_count |
| ------- | -------------- |
| US      |      3,216,292 |
| JP      |         55,160 |
| CN      |         34,118 |
| GB      |         24,925 |
| IN      |          8,064 |
| None    |          7,190 |
| FR      |          6,131 |
| DE      |          5,703 |
| IL      |          5,109 |
| ES      |          4,710 |

dmtucker · 2018-03-06T23:34:59Z

Notes

Switching PyPI to HTTPS-only should mean that occurrences dropped in Oct/Nov 2017 (but they didn't).

urllib.urlretrieve uses FancyURLopener which inherits from URLopener which sets the User-Agent.

easy_install's user-agent logic:
https://github.com/pypa/setuptools/blob/97ff22f31ace57f4eabb6f1e77c9c553de0d1c24/setuptools/package_index.py#L50

log parser user agent logic:
https://github.com/pypa/linehaul/blob/master/linehaul/user_agents.py

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 365 --where 'details.python = "1.17"' '' system distro
Served from cache: False
Data processed: 146.60 GiB
Data billed: 146.60 GiB
Estimated cost: $0.72

| system_name | distro_name | download_count |
| ----------- | ----------- | -------------- |
| None        | None        |      3,405,944 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 365 --where 'details.python = "1.17"' '' installer installer-version
Served from cache: False
Data processed: 153.04 GiB
Data billed: 153.04 GiB
Estimated cost: $0.75

| installer_name | installer_version | download_count |
| -------------- | ----------------- | -------------- |
| None           | None              |      3,405,944 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 365 --where 'details.python = "1.17"' '' impl impl-version
Served from cache: False
Data processed: 155.91 GiB
Data billed: 155.91 GiB
Estimated cost: $0.77

| implementation | impl_version | download_count |
| -------------- | ------------ | -------------- |
| None           | None         |      3,405,944 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 365 --where 'details.python = "1.17"' '' setuptools-version
Served from cache: False
Data processed: 51.03 GiB
Data billed: 51.03 GiB
Estimated cost: $0.25

| setuptools_version | download_count |
| ------------------ | -------------- |
| None               |      3,405,944 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 365 --where 'details.python = "1.17"' '' openssl
Served from cache: False
Data processed: 173.99 GiB
Data billed: 173.99 GiB
Estimated cost: $0.85

| openssl_version | download_count |
| --------------- | -------------- |
| None            |      3,405,944 |

Python Version Trends

None of these seem to correspond with 1.17 trends:

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 1095 --limit 36 --order download_month --where 'details.python = "2.6"' '' month
Served from cache: False
Data processed: 186.30 GiB
Data billed: 186.30 GiB
Estimated cost: $0.91

| download_month | download_count |
| -------------- | -------------- |
| 2018-03        |        146,529 |
| 2018-02        |        828,609 |
| 2018-01        |      1,141,411 |
| 2017-12        |      1,614,495 |
| 2017-11        |      1,371,796 |
| 2017-10        |      2,023,681 |
| 2017-09        |      2,289,735 |
| 2017-08        |      2,704,477 |
| 2017-07        |      2,828,229 |
| 2017-06        |      2,889,239 |
| 2017-05        |      2,043,001 |
| 2017-04        |      2,457,703 |
| 2017-03        |      2,680,095 |
| 2017-02        |      2,576,469 |
| 2017-01        |      3,066,075 |
| 2016-12        |      3,053,485 |
| 2016-11        |      3,591,192 |
| 2016-10        |      3,270,814 |
| 2016-09        |      3,209,299 |
| 2016-08        |      4,005,600 |
| 2016-07        |      3,946,704 |
| 2016-06        |      4,171,922 |
| 2016-05        |      2,034,229 |
| 2016-03        |      1,815,018 |
| 2016-02        |      9,519,408 |
| 2016-01        |      2,959,021 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 1095 --limit 36 --order download_month --where 'details.python = "2.7"' '' month
Served from cache: False
Data processed: 186.30 GiB
Data billed: 186.30 GiB
Estimated cost: $0.91

| download_month | download_count |
| -------------- | -------------- |
| 2018-03        |      4,668,812 |
| 2018-02        |     22,073,881 |
| 2018-01        |     30,490,386 |
| 2017-12        |     31,818,101 |
| 2017-11        |     30,530,294 |
| 2017-10        |     42,417,489 |
| 2017-09        |     47,877,172 |
| 2017-08        |     61,447,951 |
| 2017-07        |     63,732,403 |
| 2017-06        |     66,186,931 |
| 2017-05        |     68,284,527 |
| 2017-04        |     71,634,346 |
| 2017-03        |     75,668,717 |
| 2017-02        |     72,209,408 |
| 2017-01        |     80,589,481 |
| 2016-12        |     71,611,590 |
| 2016-11        |     78,672,198 |
| 2016-10        |     68,610,169 |
| 2016-09        |     74,346,875 |
| 2016-08        |     72,597,214 |
| 2016-07        |     62,769,844 |
| 2016-06        |     50,753,857 |
| 2016-05        |     17,155,656 |
| 2016-03        |      9,538,368 |
| 2016-02        |     48,248,300 |
| 2016-01        |     15,114,515 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 1095 --limit 36 --order download_month --where 'details.python = "3.4"' '' month
Served from cache: False
Data processed: 186.30 GiB
Data billed: 186.30 GiB
Estimated cost: $0.91

| download_month | download_count |
| -------------- | -------------- |
| 2018-03        |        453,018 |
| 2018-02        |      1,966,827 |
| 2018-01        |      2,435,945 |
| 2017-12        |      2,517,948 |
| 2017-11        |      2,310,739 |
| 2017-10        |      2,491,198 |
| 2017-09        |      2,332,760 |
| 2017-08        |      2,697,636 |
| 2017-07        |      2,397,228 |
| 2017-06        |      1,834,341 |
| 2017-05        |      1,863,789 |
| 2017-04        |      1,848,761 |
| 2017-03        |      2,238,398 |
| 2017-02        |      1,794,172 |
| 2017-01        |      1,793,060 |
| 2016-12        |      1,519,826 |
| 2016-11        |      1,539,866 |
| 2016-10        |      1,660,522 |
| 2016-09        |      1,602,485 |
| 2016-08        |      1,999,833 |
| 2016-07        |      1,976,705 |
| 2016-06        |        792,186 |
| 2016-05        |        363,550 |
| 2016-03        |        188,615 |
| 2016-02        |        977,270 |
| 2016-01        |        258,988 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 1095 --limit 36 --order download_month --where 'details.python = "3.5"' '' month
Served from cache: False
Data processed: 186.30 GiB
Data billed: 186.30 GiB
Estimated cost: $0.91

| download_month | download_count |
| -------------- | -------------- |
| 2018-03        |        419,007 |
| 2018-02        |      2,124,505 |
| 2018-01        |      2,331,339 |
| 2017-12        |      2,649,399 |
| 2017-11        |      2,759,223 |
| 2017-10        |      2,312,185 |
| 2017-09        |      2,016,890 |
| 2017-08        |      2,438,073 |
| 2017-07        |      2,391,372 |
| 2017-06        |      3,384,902 |
| 2017-05        |      2,832,727 |
| 2017-04        |      3,037,491 |
| 2017-03        |      2,466,478 |
| 2017-02        |      2,367,826 |
| 2017-01        |      2,417,595 |
| 2016-12        |      2,464,993 |
| 2016-11        |      1,820,581 |
| 2016-10        |      2,006,853 |
| 2016-09        |      1,761,309 |
| 2016-08        |      1,242,226 |
| 2016-07        |      1,004,178 |
| 2016-06        |        603,266 |
| 2016-05        |        333,189 |
| 2016-03        |         89,432 |
| 2016-02        |        489,781 |
| 2016-01        |        105,938 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 1095 --limit 36 --order download_month --where 'details.python = "3.6"' '' month
Served from cache: False
Data processed: 186.30 GiB
Data billed: 186.30 GiB
Estimated cost: $0.91

| download_month | download_count |
| -------------- | -------------- |
| 2018-03        |        460,217 |
| 2018-02        |      2,271,910 |
| 2018-01        |      2,236,416 |
| 2017-12        |      2,251,264 |
| 2017-11        |      5,606,128 |
| 2017-10        |      2,226,206 |
| 2017-09        |      2,054,285 |
| 2017-08        |      2,339,605 |
| 2017-07        |      1,661,571 |
| 2017-06        |      1,726,659 |
| 2017-05        |      1,152,307 |
| 2017-04        |        902,785 |
| 2017-03        |        792,730 |
| 2017-02        |        580,227 |
| 2017-01        |        391,527 |
| 2016-12        |         71,144 |
| 2016-11        |         26,199 |
| 2016-10        |         23,336 |
| 2016-09        |         23,020 |
| 2016-08        |         23,118 |
| 2016-07        |         20,472 |
| 2016-06        |         10,369 |
| 2016-05        |          3,076 |
| 2016-03        |          1,319 |
| 2016-02        |          7,425 |
| 2016-01        |          1,597 |

dmtucker/1.17

dmtucker commented Mar 4, 2018 •

edited

Loading

dmtucker commented Mar 4, 2018 •

edited

Loading

dmtucker commented Mar 5, 2018 •

edited

Loading

dmtucker commented Mar 5, 2018 •

edited

Loading

dmtucker commented Mar 6, 2018

dmtucker commented Mar 6, 2018 •

edited

Loading

dmtucker/1.17

dmtucker commented Mar 4, 2018 • edited Loading

1.17 Discussions

March 4, #python on Freenode

March 5, #python on Freenode

dmtucker commented Mar 4, 2018 • edited Loading

What is 1.17 used for?

dmtucker commented Mar 5, 2018 • edited Loading

How long has 1.17 been showing up?

dmtucker commented Mar 5, 2018 • edited Loading

Where does 1.17 come from?

dmtucker commented Mar 6, 2018

Where is 1.17 being used from?

dmtucker commented Mar 6, 2018 • edited Loading

Notes

Python Version Trends

dmtucker commented Mar 4, 2018 •

edited

Loading

March 4, `#python` on Freenode

March 5, `#python` on Freenode

dmtucker commented Mar 4, 2018 •

edited

Loading

dmtucker commented Mar 5, 2018 •

edited

Loading

dmtucker commented Mar 5, 2018 •

edited

Loading

dmtucker commented Mar 6, 2018 •

edited

Loading