Copy over GitHub subdirectory to new repo including git history

gh-copy-subdirectory-to-new-repo-and-include-history.md

Background

The following is a step by step guide for copying over a subdirectory of an existing GH repository into a new repository, which also includes the git history.

Steps

1) Clone the source repo

NOTE: The tool we will be using in step 2 requires a fresh clone. If you want to use a specific version, you can run command in step 2 with a --force command.

git clone  https://github.com/replace-with-source-repo.git
cd replace-with-source-repo

2) Filter the Subdirectory

We will use the git filter-repo tool to filter to the subdirectory you want to copy over. This will allow us to copy over only the relevant history. If you don't have git filter-repo installed, you can install it following the instructions here.

Run the following command within your source repo:

git filter-repo --subdirectory-filter path/to/subdirectory --tag-rename '':'subdir-'

This command will rewrite the history to only include the specified subdirectory. The --tag-rename option ensures that tags are renamed to avoid conflicts. You will notice that the current directory now points to the contents within the subdirectory noted above and all other files are now removed. This is why the tool prefers using a freshly cloned repo so as not to disturb any work in progress environments.

Running the following commands will ensure we cleanup any dangling objects leftover when the filtered branch is created:

git reflog expire --expire=now --all
git gc --prune=now --aggressive

3) Check commit signatures

This is a required step for Etsy repositories and may be lost when filtering the repository in the prior steps.

Run the following command to check the signatures in the log.

git log --show-signature

You are looking for a signature block right under the commit message, such as:

commit <hash>
gpg: Signature made Tue Dec 10 10:04:28 2024 EST
gpg:                using EDDSA key <key>
gpg: Good signature from "Yasmary Diaz <[email protected]>" ...
... rest of commit message ...

If there are commits that do not have a signature block, following these steps to resolve the issue:

1. To re-sign all commits, run command: git rebase -i --exec 'git commit --amend --no-edit -S' --root OR for a certain number of commits: git rebase -i --exec 'git commit --amend --no-edit -S' HEAD~<number-of-commits>
   • This will sign all the commits with your signature. This will result in showing there were two authors for a commit in the history.
2. Git will re-sign each commit during the rebase process. If there are any conflicts, resolve them and continue the rebase with command: git rebase --continue
3. Verify the log signatures by running command: git log --show-signature

4) Create a New Repository (if needed)

If you haven't already created the target repository, do so now. You can create a new GH repo using TF. Once repository is available, clone it locally.

git clone https://github.com/replace-with-target-repo.git
cd replace-with-target-repo

5) Add the Filtered Repository as a Remote

From within the new target repository, we want to add a pointer to our filtered local source repository.

git remote add local-source ../replace-with-source-repo

6) Pull the files into the target repository

We now want to fetch the changes from the filtered repository and merge them into your target repository. By using the --allow-unrelated-histories option, you override this default behavior and instruct Git to merge the histories despite the lack of a common ancestor.

git fetch local-source
git merge local-source/main --allow-unrelated-histories

NOTE: If you created your repository in the previous step using TF, it will create a README.md file which is a conflict you may have to resolve.

git add <files-merged>
git commit -S

This will prompt you to create a merge commit.

7) Clean Up

Remove the remote reference to the filtered repository.

git remote remove local-source

8) Create a branch

If you created the repo using TF or following etsy best practices, you will need to push the changes via a PR. Create a branch if you haven't already done so.

git checkout -b replace-with-branch-name

9) Open PR

Once changes are ready, let's push our changes upstream.

git push --set-upstream origin replace-with-branch-name

Open a PR. The PR will contain all prior commits applicable to the folders we filtered to. The commits will also have their corresponding signatures.

If you anticipate any conflicts may arise (particularly with the README.md), you should rebase from main to prepare for the next step. Run the following commands:

git fetch origin
git rebase origin/main

Resolve your conflicts manually, then:

git add <conflicted-files>
git rebase --continue
git push -f

or Resolve all conflicts using your current changes (the updates you are pulling in), with:

grep -lr '<<<<<<<' . | xargs git checkout --ours
git add .
git commit -S
git rebase --continue
... repeat above if there are more files
git push -f

10) Merging the PR

NOTE: You will want to make sure that the rebase and merge PR merge config option is turned on for your repo. This will ensure that all of your commits are written to the main branch when the PR is merged. Otherwise, if you use squash and merge it will result in one commit and your history will be lost.

Once PR is merged (using rebase and merge option in the dropdown), all files and history are now available in the new repo!

If we wanted to add more changes from old repo to new repo after these steps are completed, could we use step 5 in the same way?

If we wanted to add more changes from old repo to new repo after these steps are completed, could we use step 5 in the same way?

Yes that is correct! You could reuse steps 5-7 in the event that you removed the remote reference.

Updated the gist to include how to ensure that the commits are signed when bringing them over to the new repo.