A small bash script to migrate Maven packages from Sonatype Nexus to GitHub Packages

migrate-nexus-to-github-packages.md

Migrate Maven packages from Sonatype Nexus to GitHub Packages

This gist describes the process we've used to migrate our Maven package repository from Sonatype Nexus to GitHub Packages. The same process could be used for migrating any Maven package repo that is in the standard layout.

We created a special repository on GitHub to hold all of our Maven packages. You might decide to migrate packages to different repositories, in which case invoke the script multiple times.

The script uses find to look for all of the folders containing poms and upload them. You specify the folder to start from, so you upload part of your repository.

A hidden .migrated-github-packages file is created in each folder that's processed so you can interrupt and repeat the process.

1. Create the maven-releases GitHub repository as a private (or public) repository.
2. Create a personal access token for your GitHub account with permissions: repo and write:packages.
3. Create or edit your ~/.m2/settings.xml to include a <repository> and <server> element as in the example file, substituting your username and personal access token.
4. Run the script migrate-nexus-to-github-packages.sh /srv/nexus/storage/releases <owner> maven-releases github

It is possible to paralellise the script, as GitHub is pretty slow at accepting each package (~30s / package). I recommend parallelising by artifact, as Maven isn't thread safe in the way it writes to its files. I achieved that quite simply by spawning individual scripts, one for each artifact, and running them in the background. Beware that Maven actually uses a bit of memory and CPU, so doing too many artifacts in parallel might overload your system.

migrate-nexus-to-github-packages.sh

#!/bin/bash -eu

basedir="${1:-}" # The directory to start looking for poms from
owner="${2:-}" # The owner username on GitHub
repository="${3:-}" # The repository name on GitHub
serverid="${4:-}" # Matches to <server> in ~/.m2/settings.xml

if [ -z "$basedir" -o -z "$owner" -o -z "$repository" -o -z "$serverid" ]; then
  echo "usage: $0 <basedir> <owner> <repository> <serverid>" >&2
  exit 1
fi

trap ctrl_c INT

ctrl_c() {
	echo "Interrupted" >&2
	exit 1
}

# Find poms in artifact and version order
for pom in $(find "$basedir" \( -name '.nexus' \) -prune -false -o -name '*.pom' | sort --version-sort) ; do
	dir=$(dirname $pom)
	dir=$(cd "$dir" && pwd)
	if [ -f "$dir/.migrated-github-packages" ]; then
		continue
	fi

	version=$(basename $dir)
	artifact=$(basename $(dirname $dir))
	jar="$dir/$artifact-$version.jar"
	sources="$dir/$artifact-$version-sources.jar"
	javadoc="$dir/$artifact-$version-javadoc.jar"
	pomfile="$dir/$artifact-$version.pom"

	command="mvn -q org.apache.maven.plugins:maven-deploy-plugin:3.0.0-M1:deploy-file \
		-DrepositoryId=$serverid \
		-Durl=https://maven.pkg.github.com/$owner/$repository"

	if [ -f "$jar" ]; then
		command="$command -Dfile=\"$jar\""
	elif [ -f "$pomfile" ]; then
		command="$command -Dfile=\"$pomfile\""
	fi
	if [ -f "$sources" ]; then
		command="$command -Dsources=\"$sources\""
	fi
	if [ -f "$javadoc" ]; then
		command="$command -Djavadoc=\"$javadoc\""
	fi
	if [ -f "$pomfile" ]; then
		command="$command -DpomFile=\"$pomfile\""
	fi

	echo "$pomfile"

	eval $command
	touch "$dir/.migrated-github-packages"
done

settings.xml

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      http://maven.apache.org/xsd/settings-1.0.0.xsd">

  <activeProfiles>
    <activeProfile>github</activeProfile>
  </activeProfiles>

  <profiles>
    <profile>
      <id>github</id>
      <repositories>
        <repository>
          <id>central</id>
          <url>https://repo1.maven.org/maven2</url>
          <releases><enabled>true</enabled></releases>
          <snapshots><enabled>true</enabled></snapshots>
        </repository>
        <repository>
          <id>github-REPOSITORY</id>
          <name>GitHub OWNER Apache Maven Packages</name>
          <url>https://maven.pkg.github.com/OWNER/REPOSITORY</url>
          <releases><enabled>true</enabled></releases>
          <snapshots><enabled>false</enabled></snapshots>
        </repository>
      </repositories>
    </profile>
  </profiles>

  <servers>
    <server>
      <id>github</id>
      <username>USERNAME</username>
      <password>PERSONAL_ACCESS_TOKEN</password>
    </server>
  </servers>
</settings>

@karlvr : ✨ for writing this up!

Out of curiosity, did you look into any other tools such as https://github.com/microsoft/mvnfeed-cli for migration purposes?

@andyfeller I didn't see that tool, no. In the end I quite enjoyed knowing exactly what was happening with my precious jars and poms :-)

Thanks so much for sharing this. I modified it to migrate from Nexus to CodeArtifact and it works great 👍. I had trouble getting mvnfeed working, and I also like to know what's happening with my precious jars and poms.

@davidmerrick Would you mind sharing the changes that you made? It seems like it's exactly what I need.

Thank you for this! Very handy to help us get some Scala packages out of JFrog Artifactory (with jfrog rt dl "maven/*") and into GitHub Packages. 🚀

Hey @jazeph, so sorry--I'm just now seeing your comment. I'm trying to recall what I did. iirc, I got an STS token from AWS that gave me access to CodeArtifact and used that for the creds. I'll dig through my local repos and if I find more info I'll add it here.

Hey @jazeph, so sorry--I'm just now seeing your comment. I'm trying to recall what I did. iirc, I got an STS token from AWS that gave me access to CodeArtifact and used that for the creds. I'll dig through my local repos and if I find more info I'll add it here.

No worries @davidmerrick, I did a similar thing and managed to get it working. I do remember that I ran into some problems but don’t remember which ones and eventually I could work around them.

Hey @jazeph, so sorry--I'm just now seeing your comment. I'm trying to recall what I did. iirc, I got an STS token from AWS that gave me access to CodeArtifact and used that for the creds. I'll dig through my local repos and if I find more info I'll add it here.

No worries @davidmerrick, I did a similar thing and managed to get it working. I do remember that I ran into some problems but don’t remember which ones and eventually I could work around them.

Oh good! I'm glad to hear that. Cheers!