The project is to provide infrastructure to OpenWrt for CI-style automated builds. That is, the user provides a source repository that is not build by upstream OpenWrt packages, and they get a binary repository that they can use to install the packages from their source repository directly on their router, so that, for example, they can easily add a firewall, DHCP, or other packages that is not otherwise provided by the official OpenWrt repositories.
The project has been able to produce automated CI builds for both GitHub and GitLab, and include automated testing for different architectures through Docker's cross-architecture "platform" feature. The main task left is to test the binaries on real hardware, though unfortunately my router does not support OpenWrt due to binary blobs.
Freifunk gave GSoC students the ability to write blog posts as part of their project on their WordPress site. This has been very useful for tracking the overall state of the project:
GSoC ’23: Documenting the OpenWrt compilation process to set up a PPA
The first task I have tackled was to learn about the OpenWrt compilation process so I could find common operations that are always done when building different packages. I learned about how OpenWrt emulates GNU Make and uses it to produce a custom setup where multiple packages can be build from the same buildroot. This is also when I started reading and working through code produced by OpenSUSE in OpenBuildService so that I can port OpenWrt to it, following the original plan to set up CI in this way.
GSoC ’23 OpenWrt PPA Part 2: GitLab packaging
In this next blog post, we had to adjust the direction of the project. We discovered that OpenBuildService requires no networking during the build and so all packages must be provided separately to maintain reproducibility. Because OpenWrt is not yet designed around reproducible builds, and we did not want to have end-users maintain tarballs or other complex tasks to produce reproducible builds so another approach was taken.
Instead I started from my mentor Zoobab's example repository which used GitLab for automated builds and I added repository hosting through GitLab's support for generic packages and also automated testing through OpenWrt's rootfs Docker images. This allows the user to be confident that the build successfully runs and produces binaries that can run given ABI compatibility between their router's version of OpenWrt and the rootfs version in Docker.
GSoC ’23: OpenWrt PPA Porting to GitHub and Rebuilding
In this third blog post, I port the GitLab CI version to GitHub. Fortunately, the script was really nice and portable and only needed some syntax to handle. The one special case was where to place the binaries; placing them on GitLab would not make a lot of sense. Fortunately, GitHub releases works perfectly for this job so I have set the build to upload to releases which simplified the work a lot.
This is when I discover that opkg (OpenWrt's package manager) allows arbitrary file paths so I can now use this to simplify the download structure. This means that the router no longer needs an API token to download the packages. This cleans up both the code and the burden on the end user when it comes to actually install the package produced on the router, as now all that is needed is copying an HTTP URL into the opkg repo configuration.
I then work on updating the SDK which required new packages that were not yet available in the Docker Alpine development setup. This required upgrading the Alpine version, which broke compatibility with python2 but fortunately this was no longer required by newer versions of the OpenWrt SDK.
This is the initial pull request to the OpenBuildService. As we discover that the official instance of OBS does not allow networking, we discover we need to take a different approach. Of course, this PR does demonstrate how an OpenWrt build could look like in Perl, so this could be looked at further by future GSoC contributors.
GSoC '23: Add support for repository uploads and automated CI testing
In this pull request I add support for automated repository uploads and CI testing to Zoobab's GitLab repository. This allows code to be sent through to a repository and then tested, all through standardised GitLab features. Though figuring out the specifics was complicated, the final patch is quite short at a few hundred lines.
This fork of openwrtsdk updates the SDK builds and adds compatibility with newer versions of the SDK, as described when working out the python2->3 compatibility changes.
This is the GitLab port of the GitHub CI setup. Comparing the code shows that the differences are minimal so improvements to one code base can easily translate to the other, so users that prefer GitLab or GitHub can get the same feature improvements and bug fixes over time.
This tiny project demonstrates how to trigger a GitLab pipeline remotely through a browser. This can help in cases when the builds are nondeterministic or when creating a CI setup that is done with manually triggered builds.