Maintaining a personal ZMK fork

git_for_zmk.md

A cookbook approach to maintaining a personal ZMK fork

This is a brief introduction to the tools needed to maintain a personal fork of ZMK (or QMK or really whatever). It covers:

1. the initial setup
2. updating your fork with the latest ZMK features
3. merging PRs into your fork
4. deleting PRs from your fork
5. resetting your fork
6. using your fork to compile with Github Actions
7. appendix: glossary

Throughout I am using examples from ZMK but nothing is specific to ZMK (other than the compiling stuff). I am deliberately simplifying things, only covering what's absolutely essential for maintaining a personal ZMK fork for non-developers.¹

Setup

First, create your personal fork of the official ZMK repository. To do so, navigate to https://github.com/zmkfirmware/zmk and click "fork" in the upper right corner. For now your fork lives online ("remotely") on Github at https://github.com/your_username/zmk. For example mine is https://github.com/urob/zmk.

To work with your fork, next create a local "clone" of your fork on your computer. Navigate to the parent working directory in which you want to create the clone, then type:

git clone https://github.com/your_username/zmk --single-branch

The --single-branch argument is optional. It tells git to only download the main branch of your fork, which helps keeping the local clone clean.

Updating your ZMK fork with the latest ZMK

Eventually, our fork will grow out of date. To keep it updated with the latest ZMK features, we first have to register the official ZMK repo with our local clone:

git remote add -t main upstream https://github.com/zmkfirmware/zmk

Here the -t main argument is optional, it tells git to only register the "main" branch of the official ZMK repo. upstream is an alias that we will use to interact with the official ZMK repo. It can be anything, but "upstream" is customary for the source repository.

FYI, when we created our local clone, git automatically created another alias, origin, that points to our own ZMK fork on Github.

Now that we have registered the "upstream" ZMK repo, we can use it to update our own ZMK fork:

git fetch upstream
git rebase upstream/main
git push --force

The "fetch" command simply checks what's new with the upstream branch. The "rebase" command resets our local ZMK branch to the latest upstream state and then re-adds everything that we had done in our branch. Hopefully, everything goes smoothly and there are no merge conflicts. Finally, the "push" command pushes the changes we just made in our local repo back to our remote fork on Github (the --force argument is needed here due to the way git rebase works).

Merging PRs (and other remotes) into our fork

First, we have to locate and register the remote repository that contains the branch that we want to merge. We can use the same command that we used above to register the "upstream" ZMK repo. For example, for ftc's "mouse PR" branch:

git remote add -t mouse-ftc mouse-stuff https://github.com/ftc/zmk

Again, the -t mouse-ftc is optional. Here I used it to only register the "mouse-ftc" branch (which is the one that contains the mouse PR). mouse-stuff is again an arbitrary alias -- choose what makes sense to you.²

Once we registered the remote, we can merge it into our own local clone:

git fetch mouse-stuff
git merge mouse-stuff/mouse-ftc --squash
git commit -m "Mouse emulation support"
git push

The already familiar git fetch now checks what's new with the "mouse-stuff" repository ("mouse-stuff" is the alias we chose above). git merge does the actual merging into our local clone. The --squash argument tells git to bundle all the commits in the source repo iinto a single one instead of adding them one-by-one. It's a matter of personal taste, but I like it, because it keeps my git history clean and allows for easy removal of a PR (see below). Next we commit all the merged changes with git commit using a commit message of "Mouse emulation support" (can be anything). Finally, we push these local changes back to our fork on Github.

Another example

This process can be repeated to merge as many remote branches as you like. To give another example, to add the "fix-mod-morph" PR and the "positional-hold-tap-on-release" PR, we would first register the source remote:

git remote add urob https://github.com/urob/zmk

I haven't used the -t argument this time, because I am planning to merge two branches from this repo. Once the remote is registered, we can merge the two branches and push the changes back to our remote fork with:

git fetch urob
git merge urob/fix-mod-morph --squash
git commit -m "Fix mod-morph"
git merge urob/positional-hold-tap-on-release --squash
git commit -m "On-release property for positional hold taps"
git push

Merge conflicts

Resolving merge conflicts is beyond the scope of this little introduction. While some merge conflicts are simple formatting issues, major merge conflicts require digging through the source code and understanding what's actually going on in order to resolve them correctly. Fortunately, in my experience, most PRs merge cleanly. And if not, there is often already someone who resolved the merge conflict (like ftc for the "mouse PR"), and one can just merge their branch instead.

Deleting PRs (and other commits)

Suppose at some point we decide that we no longer need the "Fix mod-morph" PR (perhaps because it has been merged into the official ZMK repo). If you followed my advice above, the entire PR is a single commit in our repo. To delete it, run:

git rebase upstream/main -i

This is almost the same command that we used above to update our repo with the latest ZMK features. As above, it first resets our local branch to the state of the upstream branch (or, more accurately, to the state it was in the last time we ran git fetch upstream). But then, because we have specified the -i argument, instead of re-applying all our local changes, it opens a text editor and lets us choose interactively what we want to do. The editor lists all our commits that we have made on top of upstream. It should look similar to this:

pick 2efdd3ce Mouse emulation support
pick d3eb8c3d Fix mod-morph
pick d3768f3a On-release property for positional hold taps

To delete one of those commits, replace "pick" by "d" and then save and exit the editor. For example, to delete the "Fix mod-morph" PR, we would do

pick 2efdd3ce Mouse emulation support
d d3eb8c3d Fix mod-morph
pick d3768f3a On-release property for positional hold taps

Once we are done, we can push back the changes to our remote fork on Github with:

git push --force

Resetting our fork

Suppose we have done something weird and screwed up our local clone. If we haven't pushed our changes back to our remote fork on Github, we can simply reset our local clone to the state of our remote fork on Github by running:

git reset --hard origin/main

Alternatively, we could reset our repo to the state of the official ZMK repo by running:

git fetch upstream
git reset --hard upstream/main
git push --force

If we only want to undo some of our commits, but have already pushed them to our remote, we can use the interactive rebase command from above to first delete the offending commits from our local repo:

git rebase upstream/main -i

Once we fixed our local repo, we can then push back the repaired repo to our remote fork on Github:

git push --force

Using our own fork to compile with Github actions

Now that you have your shiny own personalized ZMK fork, you want to use it to build the firmware. You could just embrace the full experience, install the toolchain, and compile locally. But perhaps you want to save the full experience for another day. In this case, you can still use your own repo to build with Github Actions by replacing the west.yml file in your zmk-config repo (not the same as your new zmk repo!) with this:

manifest:
  remotes:
    - name: some_name
      url-base: https://github.com/yourusername
  projects:
    - name: zmk
      remote: some_name
      revision: main
      import: app/west.yml
  self:
    path: config

Conclusion

That's it. You now know how to update your fork with the latest official features, how to merge PRs into your fork or delete them, and how to reset your repo. Happy building!

Glossary

Okay, I get it, there's a lot of terms flying around. This is what some of them mean:

• remote: A repository that lives somewhere online. In the guide above we have used three types of remotes:
  • upstream: the original source repository (https://github.com/zmkfirmware/zmk/)
  • origin: your personal fork on Github (https://github.com/your_username/zmk/)
  • other remotes with branches that we did merge (https://github.com/some_dude/zmk/)
• our local clone: a local copy of origin

Note that there is no automatic syncronization in git. To make our local clone aware of new stuff happening in any remote, we use git fetch. To actually apply changes from a (fetched) remote to our local clone, we use git merge and git rebase (there's also git pull and git cherry-pick). And to push back changes from our local clone to origin (or other remotes with write-access), we use git push.

Footnotes

1. This means that I am leaving out many core concepts such as git add, git pull, git branch (and checkout), git stash, git submodule and git subtree, git cherry-pick, resolving merge-conflicts, viewing diffs and logfiles, etc. The reason is that there are already tons of great introductions to git. The problem is that git is complex, which can make even the best introductions a bit daunting, especially for people who only want to combine a couple of PRs without developing their own code (if you develop your own code, you probably don't need help using git). Hence the idea for this "cookbook approach". ↩

2. Personally I like to use remote aliases based on the maintainers username. But for the purpose of this guide, I wanted to emphasize that it is the alias that it used when interacting with the repo, not the username. Hence, my choice of "mouse-stuff" as opposed to "ftc". ↩

It's a nice summary of the topic. I think it's worth adding a description how to manage contributing back into the upstream repos.
So for example if I have my repo that combines multiple branches from various upstream repos and I want a modification I made to go back as PR to one of those repos, how should it be done so I can isolate only that change without all the extra modifications from other upstream repos I've incorporated into mine.

It's a nice summary of the topic. I think it's worth adding a description how to manage contributing back into the upstream repos. So for example if I have my repo that combines multiple branches from various upstream repos and I want a modification I made to go back as PR to one of those repos, how should it be done so I can isolate only that change without all the extra modifications from other upstream repos I've incorporated into mine.

An exhaustive guide to creating PRs would be a bit beyond the scope of this introduction, which purposefully is kept as simple and the least daunting as possible. I would think that most people developing new features are already somewhat familiar with Git or aren't scared away by the many more comprehensive guides out there.

But basically, you'd want to create a separate branch that isolates the features of the PR. There are many ways to get there; e.g., when the PR only includes a few commits, it is often most effective to branch off upstream and use git cherry-pick to copy over the relevant commits. But since I haven't touched on that at all, let me suggest another way here using git rebase that we already introduced in the guide above.

Essentially, the idea is to create a new branch that includes more than what we want to include in the PR, and then do an interactive rebase to throw out all the unwanted stuff. To begin, we need to create and checkout a new branch that includes all the commits we want (and possibly more). If they live in the currently checked-out branch (e.g., your main branch), we can do both in one step using:

git checkout -b my_pr_branch

Here, my_pr_branch is an arbitrary name given to the new branch. Once we are in the PR branch, we can simply do an interactive rebase onto the latest upstream branch, select all the commits that we want to include, and push the new branch to our Github repository:

git fetch upstream
git rebase -i upstream/main
git push -u origin my_pr_branch

As we did throughout the guide, we precede the rebase with a git fetch to make sure we are rebasing on the latest version of upstream (or the local alias of whichever repo we want to contribute to). The final step then pushes our new branch, which so far only exist locally, to our remote repo on Github (aka "origin"). Note that my_pr_branch must match the name that we have given our local PR-branch above. Once the new branch is pushed to Github, we can open a PR through the Github interface.

Thanks for this 👍 I use Git daily and you still managed to explain this in such a clear and concise way that I learned a thing out two.

I followed this carefully from a clean installation, all steps worked until this one, getting this error on the merge step:

$ git merge mouse-stuff/mouse-pr --squash
merge: mouse-stuff/mouse-pr - not something we can merge

Is it something I am doing incorrectly or is this because ZMK has changed since this was written?

I followed this carefully from a clean installation, all steps worked until this one, getting this error on the merge step:

$ git merge mouse-stuff/mouse-pr --squash
merge: mouse-stuff/mouse-pr - not something we can merge

Is it something I am doing incorrectly or is this because ZMK has changed since this was written?

Two things:

1. The purpose of this gist is to give an introduction to how git works. The purpose is not to document how to get mouse keys or any other specific functionality to work with ZMK. While the examples chosen in the gist were relevant at the time of writing, those branches are all outdated now and will likely result in merge conflicts.
2. That said, based on your error message, it doesn't look like a merge conflict is the issue here. The error message is consistent with the alias "mouse-stuff" not resolving. Are you sure you followed all steps prior to the git merge command, including the remote add command where I define mouse-stuff?

remote add appears to work correctly:

$ git remote add -t mouse-ftc mouse-stuff https://github.com/ftc/zmk
error: remote mouse-stuff already exists.

$ git fetch mouse-stuff

$ git merge mouse-stuff/mouse-pr --squash
merge: mouse-stuff/mouse-pr - not something we can merge

remote add appears to work correctly:

$ git remote add -t mouse-ftc mouse-stuff https://github.com/ftc/zmk
error: remote mouse-stuff already exists.

$ git fetch mouse-stuff

$ git merge mouse-stuff/mouse-pr --squash
merge: mouse-stuff/mouse-pr - not something we can merge

Try

git merge mouse-stuff/mouse-ftc --squash

The branch mouse-pr does not exist in ftc's repo -- the correct branch name is mouse-ftc (the same one you registered with remote add two commands prior)

Yes, thank you! I swear I can read! 😆

Hi, do you have any experience with building the firmware with Docker or west toolchain? I'm using this Docker approach with success to build (ZMK docs provide instructions that involve devcontainers/VS Code and I wanted a more general approach). In the Docker container, my west.yml pulls the projects Urob's fork of ZMK ank zmk-helpers into the container:

manifest:
  remotes:
    - name: urob
      url-base: https://github.com/urob
  projects:
    - name: zmk
      remote: urob
      revision: main
      import: app/west.yml
    - name: zmk-helpers
      remote: urob
      revision: v2
  self:
    path: config

and my zmk-config (based on Urob's) is accessible to the container via Docker volume mount. Now I'm interested in using keymap-drawers but it needs to reference/access zmk-helpers/include. Thus far, I don't have a local copy of Urob's zmk fork, zmk-helpers, and keymap-drawers. Do I want a local copy of any of these? Should I be installing pipx (needed for keymap-drawer) and keymap-drawer into the container so that it can access zmk-helpers/include that gets pulled by west and the keymap image can be produced together automatically every time the firmware is updated? Else I can keep a local copy of zmk-helpers in my zmk-config (as a git submodule I suppose) and also keep keymap-drawers installed locally to run manually after make builds the firmware.

My original intention was to use the Docker approach to do all zmk-related stuff and not pollute my host system with unnecessary dependencies (e.g. the toolchain dependencies). AFAIK keymap-drawer installed via pipx means it's in a virtual environment so can be considered self-contained and therefore not a concern for my purposes(?). I'm not sure of the best way to structure all zmk-related stuff, e.g. keymap-drawer will only ever be used for anything zmk-related, so maybe I want it in the repo somehow (as a binary? Is that recommended?). But my zmk-config is on the host system where I do the development so this is probably a pointless endeavor.

Appreciate any advice, e.g. whether keymap-drawer should be pulled by west.yml and installed in the container.

P.S. Docker-related questions:

• What's the point of make clean_image in the Makefile above? If I understand correctly, with make, the image is cached until there is an update to the image.

• What's the point of building the firmware in an "ephemeral container" as in docker --rm in the Makefile? Why not re-use the same container?

Much appreciated.

Now I'm interested in using keymap-drawers but it needs to reference/access zmk-helpers/include.

Just a few quick pointers since this goes beyond the purpose of this git.

You have a couple of options here.

Option 1: Docker

For this, you'd need to install keymap-drawer to the same docker image. The tricky part is that the linked PR isn't yet part of the released version. So you'd need to use a custom build for keymap-drawers (or wait a bit and use pipx).

In either case, the container should already pull in a copy of zmk-helpers when running west. So getting keymap-drawers to work is only a question of finding the path where west pulls in zmk-helpers and correctly configuring the zmk_additional_includes option for keymap-drawers.

Option 2: Run locally

This is probably the easiest, especially if you have a local copy of zmk-helpers. If the copy resides inside your zmk-config, keymap-drawers should just work as is. Otherwise, if you use it as west module, you need a version of keymap-drawers with PR 105 and reference the path of zmk-helpers per its documentation.

Option 3: Let Github-Actions do it

There's a sister-PR to keymap-drawers which let's you use modules with the Github Actions workflow. @stijnveenman has it all up running in their zmk-config if you want to check it out.