Comprehensive Python testing on Travis CI

.travis.yml

language: python
# ===== Linux ======
os: linux
dist: xenial
python:
  - 2.7
  - 3.6
  - 3.7
  - 3.8
  - 3.9
jobs:
  include:      
    # ======= OSX ========
#     ----- changes in Travis images means this doesn't work for versions before 3.7.5 ---
#     - name: "Python 2.7.14 on macOS 10.13"
#       os: osx
#       osx_image: xcode9.3  # Python 2.7.14_2 running on macOS 10.13
#       language: shell       # 'language: python' is an error on Travis CI macOS
#       before_install:
#         - python --version
#         # - pip install -U pip
#         # - python -m pip install --upgrade pip
#         - pip install pytest --user
#         - pip install codecov --user
#       install: pip install ".[test]" --user
#       script: python -m pytest
#       after_success: python  -m codecov
#     - name: "Python 3.6.5 on macOS 10.13"
#       os: osx
#       osx_image: xcode9.4  # Python 3.6.5 running on macOS 10.13
#       language: shell       # 'language: python' is an error on Travis CI macOS
#       before_install:
#         - python3 --version
#         - pip3 install -U pip
#         - pip3 install -U pytest
#         - pip3 install codecov
#       script: python3 -m pytest
#       after_success: python3 -m codecov
    - name: "Python 3.7.5 on macOS 10.14"
      os: osx
      osx_image: xcode10.2  # Python 3.7.5 running on macOS 10.14.3
      language: shell       # 'language: python' is an error on Travis CI macOS
      before_install:
        - python3 --version
        - python3 -m pip install --upgrade pip
        - python3 -m pip install --upgrade codecov pytest
      script: python3 -m pytest
      after_success: python3 -m codecov
    - name: "Python 3.8.0 on macOS 10.14"
      os: osx
      osx_image: xcode11.3  # Python 3.8.0 running on macOS 10.14.6
      language: shell       # 'language: python' is an error on Travis CI macOS
      before_install:
        - python3 --version
        - python3 -m pip install --upgrade pip
        - python3 -m pip install --upgrade codecov pytest
      script: python3 -m pytest
      after_success: python3 -m codecov
    #  ====== WINDOWS =========
    - name: "Python 2.7 on Windows"
      os: windows           # Windows 10.0.17134 N/A Build 17134
      language: shell # 'language: python' errors Travis CI Windows
      before_install:
        - choco install python2
        - python --version
        - python -m pip install --upgrade pip
        - pip install --upgrade pytest
        - pip install codecov
      env: PATH=/c/Python27:/c/Python27/Scripts:$PATH
    - name: "Python 3.5.4 on Windows"
      os: windows           # Windows 10.0.17134 N/A Build 17134
      language: shell       # 'language: python' is an error on Travis CI Windows
      before_install:
        - choco install python --version 3.5.4
        - python --version
        - python -m pip install --upgrade pip
        - python -m pip install --upgrade codecov pytest
      env: PATH=/c/Python35:/c/Python35/Scripts:$PATH
    - name: "Python 3.6.8 on Windows"
      os: windows           # Windows 10.0.17134 N/A Build 17134
      language: shell       # 'language: python' is an error on Travis CI Windows
      before_install:
        - choco install python --version 3.6.8
        - python --version
        - python -m pip install --upgrade pip
        - python -m pip install --upgrade codecov pytest
      env: PATH=/c/Python36:/c/Python36/Scripts:$PATH
    - name: "Python 3.7.9 on Windows"
      os: windows           # Windows 10.0.17134 N/A Build 17134
      language: shell       # 'language: python' is an error on Travis CI Windows
      before_install:
        - choco install python --version 3.7.9
        - python --version
        - python -m pip install --upgrade pip
        - python -m pip install --upgrade codecov pytest
      env: PATH=/c/Python37:/c/Python37/Scripts:$PATH
    - name: "Python 3.8.8 on Windows"
      os: windows           # Windows 10.0.17134 N/A Build 17134
      language: shell       # 'language: python' is an error on Travis CI Windows
      before_install:
        - choco install python --version 3.8.8
        - python --version
        - python -m pip install --upgrade pip
        - python -m pip install --upgrade codecov pytest
      env: PATH=/c/Python38:/c/Python38/Scripts:$PATH
    - name: "Python 3.9.2 on Windows"
      os: windows           # Windows 10.0.17134 N/A Build 17134
      language: shell       # 'language: python' is an error on Travis CI Windows
      before_install:
        - choco install python --version 3.9.2
        - python --version
        - python -m pip install --upgrade pip
        - python -m pip install --upgrade codecov pytest
      env: PATH=/c/Python39:/c/Python39/Scripts:$PATH
before_install:
  - python --version
  - python3 -m pip install --upgrade pip
  - python3 -m pip install --upgrade codecov pytest
install:
  - pip install ".[test]"
script: pytest
after_success:
  - codecov # submit coverage

Would it be possible to update for Python 3.8 and osx_image: xcode11.3 (and drop legacy Python)?

Also pip will be getting a real dependency resolver soon so python3 -m pip install --upgrade codecov pytest might be better.

Well, I've added the 3.8.0 job for macOS, but it doesn't make since to remove jobs for legacy Python, since this is a template, meant for people to take and remove what they don't need. Some projects still test on legacy Python, even Python 2 after its deprecation.

About the dependency resolver - when will this happen? And why add this before this happens?

Also, thank you for the input! :)

What about Python 3.8 on Linux just after line 8?

Why prepare now for pip dependency resolution? Because it is now in beta and preparing has no negative effects but not preparing will mean that anyone who grabs this template before you change modify it will not benefit from pip having true dependency resolution.

Thanks for this awesome template 💯

---

@shaypal5 Could you have a look if the macOs python versions are still correct? I've tried your trick with the differnt osx_images but they resolve as follows:

osx_image	python3 --version
xcode9.4	Python 3.7.5
xcode10.2	Python 3.7.5
xcode11.3	Python 3.7.7

Here is a build of a sample project of mine.

Hey @pmeier,

Unfortunately, changes in Travis images means this doesn't work for versions before 3.7.5 anymore.

his used to work - Travis even put my blog post on this up on their own official blog - but I guess they keep rebuilding the images, and so now even old xcode images come with Python 3.7.5.

This is the image that gets pulled by the job, not something we can control. At the time, when I was figuring this out, this was the only way I found to easily test different Python versions on macOS. I think building it from source each time is too expensive an overhead, so we can bare to not test this on earlier Pythons specifically on macOS (they ARE tested on linux, after all).

I've encountered this for one of my projects as well:
python-cachier/cachier#23

I had to move to only testing Python 3.7.5 and Python 3.8.0 on macOS. :\

Hm, that is unfortunate. Heads up for you: even with xcode11.3 you are only testing against 3.7. See for example one of your latest builds.

---

Travis even pulled my blog post on this to their own official blog

Are you sure? I came to this gist by this

https://blog.travis-ci.com/2019-08-07-extensive-python-testing-on-travis-ci

Sorry, this was an incorrect use of American English. I'm a non-native speaker. I meant they even put it on their official blog up back in the day (August 2019), but that this has become obsolete since (I guess rather recently, but I can't tell). :)

Aren't after_success's supposed to be python3 ... instead of python 3 ... ?

• The comment on line 2 should be replaced by os: linux.
  • See View config on the Travis run page for Build config validation hints.
• matrix: on line 9 should be replaced with jobs:. matrix has been removed from Travis CI docs.
• Python 3.8 and 3.9 should be added and Python 3.5 should be dropped. Same for Windows/choco.
• pip has options for both a --upgrade and --user so the -U should be replaced with something more explicit.
• Now that pip has a real dependency resolver, once pip itself has been upgraded, all remaining dependencies should be bundled into a single command.
  • python3 -m pip install --upgrade pip
  • python3 -m pip install --upgrade codecov pytest # Allow the pip dependency resolver to do its work.

Aren't after_success's supposed to be python3 ... instead of python 3 ... ?

Definitely, that's a great catch. Thank you!
I've now fixed this.

• The comment on line 2 should be replaced by os: linux.

  • See View config on the Travis run page for Build config validation hints.

• matrix: on line 9 should be replaced with jobs:. matrix has been removed from Travis CI docs.

• Python 3.8 and 3.9 should be added and Python 3.5 should be dropped. Same for Windows/choco.

• pip has options for both a --upgrade and --user so the -U should be replaced with something more explicit.

• Now that pip has a real dependency resolver, once pip itself has been upgraded, all remaining dependencies should be bundled into a single command.

  • python3 -m pip install --upgrade pip
  • python3 -m pip install --upgrade codecov pytest # Allow the pip dependency resolver to do its work.

OK. I've amended almost everything, but someone will have to lookup the right image for Python 3.9 on OSX.