Merge conflict represents drift between main and a release branch, which is not represented in a debian/patch and is not represented in the upstream source code.
Our model of merging upstream source code into our release series branches is problematic. It involves carrying an entire copy of the upstream source code in every release branch, which inevitably drifts due to human error, and results in routine unreviewable merges during every release.
Imagine for a moment a different release model. Under this different model, the upstream branch would contain no debian/ directory (just like it is today). Downstream series release branches, however, would contain only debian/ directory. Under this different model, a release would happen by generating an orig.tar.gz directly from the tagged upstream release. To build a downstream package, one would cherry-pick the downstream debian/-only release branch onto the tagged release branch, then sbuild the result.
- no longer required to review multi-thousand-line code changes that are effectively unreviewable - no more "did my copy of the tooling do the same thing as yours? if yes, then I guess I have to assume it is correct"
- We wouldn't have to sync changes (patches, changelog) between multiple branches
- Easily build a downstream package based on any commit in main without snapshots
- Single source of truth for every file: drift & merge conflict not possible since merging changes to files is never required.
- Series release branches don't require incremental snapshots to add patches that can be built/tested
- less error prone: cant "forget to sync changelog to branch Y, which got flagged during SRU review"
This model would support:
- new series releases
- existing series point releases
- mid-cycle devel releases from arbitrary commits on main
- downstream hot fixes based on any combination of:
- upstream fix (YY.Q.N upstream release)
- downstream-only patch
- cherry-picked upstream commit into downstream
An example script that does this: https://github.com/holmanb/uss-tableflip/commit/266cac5e03317e22098ca4c8da166045367d305a
If we structure our repositories such that downstream release branches contain contain only a debian/ directory which can be rebased on upstream main branch, then generating a .deb package could be as simple as:
git clean -dx --force
git archive --format=tar.gz -9 --output=../cloud-init_${UPSTREAM_RELEASE_VERSION}.orig.tar.gz $UPSTREAM_RELEASE_COMMITTISH
# Upload this to githubgit checkout $UPSTREAM_RELEASE_COMMITTISH
git cherry-pick ..$DOWNSTREAM_RELEASE_COMMITTISH
sbuild --dist=$RELEASE_SERIES --arch=amd64 --arch-all .
# dput to LaunchpadNote that this doesn't require direct dependency on dpkg-dev or devscripts. Perhaps more interesting is that this does not require use of uss-tableflip's homegrown build-package, get-orig-tarball, or our in-tree tools/make-tarball (of which many projects apparently have their own copy which gets called by uss-tableflip tooling!). That is almost 800 lines of bash code that we just might not need. What does this tooling buy us that is worth maintaining that much bash that in theory gets ran ~4 times per year (8 if you count reviewers duplicating the release process as their "review")?
Consider the following example of building a deb package from the tip of main. This example uses an example branch named debian which contains only a debian/ directory.
$ git checkout debian
Switched to branch 'debian'
Your branch is up to date with 'origin/debian'.
$ sed -i '1s/.*/cloud-init (24.1) noble; urgency=medium/' debian/changelog
$ git commit -m "example release" debian/changelog
[debian fa618dd81] example release
1 file changed, 1 insertion(+), 1 deletion(-)
$ git checkout main
Switched to branch 'main'
Your branch is up to date with 'upstream/main'.
$ git clean -dx --force
$ git archive --format=tar.gz -9 --output=../cloud-init_24.1.orig.tar.gz HEAD
$ git cherry-pick ..debian > /dev/null
$ sbuild --dist=noble --arch=amd64 --arch-all .
$ ls -1 ../cloud-init_24.1*
../cloud-init_24.1_all.deb
../cloud-init_24.1_amd64-2024-02-14T14:37:46Z.build
../cloud-init_24.1_amd64.build
../cloud-init_24.1_amd64.buildinfo
../cloud-init_24.1_amd64.changes
../cloud-init_24.1_amd64_translations.tar.gz
../cloud-init_24.1.debian.tar.xz
../cloud-init_24.1.dsc
../cloud-init_24.1.orig.tar.gz- generate upstream changelog
- tag upstream release (24.1)
- tag downstream releases (ubuntu/noble-24.1)
- Generate orig.tar.gz
- Upload orig.tar.gz to github
- sbuild deb
- dput deb
Cloud-init team does many different things that fall into the category of "do a release". This means that there is a flow between states that sometimes has different entry points.
| Current state | Next state | Description |
|---|---|---|
| New upstream release | New downstream release | Quarterly upstream release -> Downstream release |
| Upstream point release | New downstream release | We need to fix something in the upstream release before downstream was released |
| Upstream point release | Hotfix "bump" downstream release (no downstream changes) | We already released downstream, but we need to re-release with a fix in upstream |
| Upstream point release | Hotfix "fix" downstream release (downstream changes) | We already released downstream, but we need to re-release downstream based on an upstream fix with an additional downstream patch |
| New downstream release | Hotfix "fix" downstream release | We already released downstream, but we need to re-release with a new patch for an Ubuntu-only issue |
| Any | New series | A new Ubuntu release has arrived: 24.0{4,10} |
| New series | New downstream release | First SRU release into the new series |
| New downstream release | Daily build | Not really a "release", but we need to support building "daily" packagees from tip of main |
Hotfix branches would only be required if we need to cherry pick an upstream commit from main into a downstream release: a fix that is expected to be a long lived delta from upstream would be built from a patch in the downstream release packaging branch (a new downstream packaging tag against the original upstream release), and a new downstream release based on a upstream point release would just get built using the new point release on main.
Daily builds for each series would get built from the downstream release branches, not hotfix release branches.
It's what Robbie agreed to. Obviously he's not the only voice that matters, but if you think we need less, it's worth asking Christian and/or the SRU reviewers in the server team. I can't find a hard and fast rule about what exactly is required in the changelog for upstream snapshots.
Got examples? I'm curious what they look like.
Yes, but I also want our changelog to be helpful. "Go find the upstream version on your own and then look at that version's ChangeLog" doesn't seem incredibly intuitive to me...though probably something your average person inspecting a debian/changelog would be able to do. On the other hand, I don't think a link/commit hash is really much more tooling than one to add
* New upstream version. At the end of the day, I don't have super strong opinions, but I'd lean towards keeping it.