Skip to content

Instantly share code, notes, and snippets.

@mark-kubacki
Last active April 6, 2021 21:57
Show Gist options
  • Save mark-kubacki/b42063a663db6b89a04aebd5af58baa5 to your computer and use it in GitHub Desktop.
Save mark-kubacki/b42063a663db6b89a04aebd5af58baa5 to your computer and use it in GitHub Desktop.
Collect a set of repositories, each in its own subdir, into one mono-repo.
#!/bin/bash
# Merges a set of repositories, the “sources”, formerly hosted on “from”
# into subdirectories of one repository
# meant to be moved to a new repository hosting provider.
#
# git version: 2.24.0
# Mark Kubacki, 2020-01-08
if (( $# < 1 )); then
>&2 printf "Usage: $0 [repo | repo…]\n"
exit 2
fi
set -euop pipefail
# Change this to your Gitlab, if need be. The 'slug' below will be %s.
: ${hosting:="https://source.developers.google.com/p/${USER}/r/%s"}
# This is the slug for the destination repository.
: ${D:="unrelated-containers"}
# I'm moving away from Github.
: ${leave:="[email protected]:${USER}/%s.git"}
declare -a sources=()
workdir="$(mktemp -d -t merge-repos.XXXXXX)"
cd "${workdir}"
trap "rm -rf '${workdir}'" EXIT
# Fetch all relevant repositories.
>&2 printf "==== git clone …\n"
for source; do
if [[ ! -d "${source}" ]]; then
printf -v from "${leave}" "${source}"
if ! git clone "${from}"; then
>&2 printf "ERR, skipping: %s ← %s\n" "${source}" "${from}"
continue
fi
fi
sources+=("${source}")
done
# Go through every source repository and move its contents one directory level down.
>&2 printf "==== push to subdirectories\n"
for source in "${sources[@]}"; do
cd "${source}"
FILTER_BRANCH_SQUELCH_WARNING=1 \
git filter-branch -f --prune-empty --tree-filter '
mkdir -p "'${source}'"
git ls-tree --name-only $GIT_COMMIT | xargs -I files mv files "'${source}'"
'
cd ..
done
# Pull everything into the super-repository.
>&2 printf "==== assemble the mono-repository ====\n"
printf -v destination "${hosting}" "${D}"
if git clone -o 'destination' "${destination}"; then
cd "${D}"
else
mkdir "${D}"
cd $_
git init .
git commit --allow-empty -m "Create this mono-repository"
git remote add 'destination' "${destination}"
#gcloud source repos create ${D}
fi
for source in "${sources[@]}"; do
if [[ -d "${source}" ]]; then
>&2 printf "EXISTS, skipping: %s\n" "${source}"
continue
fi
git remote add -f "${source}" ../"${source}"
git merge -s ours --allow-unrelated-histories --no-commit "${source}/master"
# No need to work with another prefix: The files already are in distinct subdirectories.
git read-tree --prefix=""/ -u "${source}/master"
git commit -m "subtree-merge: ${source}"
done
git push destination master
cd ..
# fin
exit 0
@mark-kubacki
Copy link
Author

mark-kubacki commented Jan 3, 2021

If you already have a super-repository or “target mono-repo”, you could do the following. For the sake of this example I'll call the destination directory cmd/indexer. Its git checkout is in strayrepo/:

cd strayrepo/
FILTER_BRANCH_SQUELCH_WARNING=1 \
git filter-branch -f --prune-empty --tree-filter '
 mkdir -p "cmd/indexer"
 git ls-tree --name-only $GIT_COMMIT | xargs -I files mv files "cmd/indexer/"
'

cd monorepo
git remote add -f "cmd-indexer" ../strayrepo
git merge -s ours --allow-unrelated-histories --no-commit "cmd-indexer/master"
git read-tree --prefix=/ -u "cmd-indexer/master"
git commit -m "cmd/indexer: Package from subtree-merge"

@mark-kubacki
Copy link
Author

If the source repository already contains a directory you want to move it entirely to, or below, — for example a cmd/ but everything shall go into cmd/indexer/… —, do this:

cd strayrepo/

FILTER_BRANCH_SQUELCH_WARNING=1 \
git filter-branch -f --prune-empty --tree-filter '
  mkdir -p "xmd/indexer"
  git ls-tree --name-only $GIT_COMMIT | xargs -I files mv files "xmd/indexer"
'

FILTER_BRANCH_SQUELCH_WARNING=1 \
git filter-branch --tree-filter '
  mv xmd cmd
' --force HEAD

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment