Automating a Python package release

discussion.md

Disclaimer

TL;DR: please don't spend your voluteer time helping me solve problems I'll get paid for, if you don't want the answer too.

While I'm interested in solving these questions for open source packages, the open source package I personally maintain don't have this much traction that I can't do it by myself. I'm also interested in this to solve problems I have at work, and it's disloyal to crowdsource ideas for problems I have in closed-source projects without explicitely stating so. So here I am, stating that whatever comes out of this may benefit my closed source projects maybe more than my open source ones. But maybe yours too.

The goals

• On some projects I'd like to have automatic releases after every PR merge (hypothesis style)
• On some projects, I'd like to trigger a release (either from a PR or out of the blue)

The constraints: release should:

1. Be incremental, minor or major. There should be a way to specify (ideally releated to the associated PR)
2. Have a git tag
3. Be published in PyPI
4. Have a changelog accessible somewhere (albeit no necessarily in the code. Readthedocs is ok)
5. Have a changelog containing release dates, which may not be the PR merge date
6. Have a changelog created from fragments (towncrier style) to avoid constant conflicts
7. Be a 1-step operation. I'd rather not the release process would create a PR, and this PR merge would trigger the release.

Other constraints:

8. The code should able to introspect its own current version (from mypackage import __version__), even when installed from the source in -e

9. The process should be able to run in a repository that protects master branch from commits without PR

10. And as much as possible, this should be usable in different packages, and simple to setup.

All constraints are debatable.

The problems & leads

TL;DR: I'd like a release not to necessitate a change in the code. When I release today, I have 2 changes: Towncrier changelog compilation and update of the version number in setup.cfg. How to solve both ?

Towncrier (or other fragment based changelog tool) need a "changelog compilation". This modifies the files. This means at realase time, there's a code change, so there's a commit, so there's a PR and it invalidates point 7.

We could run towncrier in the docs job only, and all the fragments and reconstruct the whole changelog from scratch everytime but we'd need a different fragment folder for each release, and to store the release date somewhere.

We could remove the fragment-based contraint, but we'd still have to figure out the release date and put that somewhere.

Then there's the problem of the current version number. Should it be written in setup.cfg ? if not and setup.cfg holds a placeholder that's only filled at release time and not committed, how to satisfy 8.

We could dynamically call git describe in this case to get the value for __version__.

What I see elsewhere

• Hypothesis: each contributors adds a RELEASE.rst file in their PR, stating patch/minor/major and the changelog. The CI releases based on this, builds the changelog, commits the removal of RELEASE.rst and pushes on master. This clashes with 9.

(found this gist via https://twitter.com/Ewjoachim/status/1147452694099050496 )

@webknjaz I bet you have some thoughts here?

As @di mentioned in the Twitter thread, the API key for uploading pypi/warehouse#6084 will definitely affect optimal workflow in the future.

@brainwane thanks for the ping!

@ewjoachim I use setuptools-scm (cc @RonnyPfannschmidt) for getting a version number from the git tree. No need to hardcode things. But if you want it's possible to run something like python -m setuptools_scm ls and it'd print that out so you could automate using that wherever you want. This plays well with (8) because it actually puts this info into dist metadata which can be consumed with pkg_resources if needed.

Overall, I think it's a case for a bot which both opens a PR and merges it, and tags the commit. It could be triggered by the button on the Checks page. If you want to try writing a GitHub bot in Python, check out https://tutorial.octomachinery.dev

If needed, the bot could orchestrate building things on the CI computing power using their APIs.

I still think that the approval part should probably be done by the human (maybe also with a button). Yet, I fully realize that adding more manual things in the flow is harmful.

Currently, I don't have any MVP but I've been meaning to play with this. So meanwhile I make continuous pre-releases to Test PyPI for every push to master (https://test.pypi.org/project/octomachinery/#history) and if a release is needed I cut a tag and that goes to a production PyPI as well.

P.S. If you've got access to GitHub Actions, you can probably use it for workflows. I'm hoping to have this https://github.com/pypa/gh-action-pypi-publish as a building block for uploading things to PyPI

Im totally interested in the proposed workflows

@RonnyPfannschmidt are you at EuroPython this year?

nope, im at home with a sick toddler

Huh... pity