Instructions on how to move a GitLab project with lots of container images in its registry

gitlab-move-project-with-containers.md

Move a GitLab project including container registry

This little post describes how to move ("transfer") a GitLab project to another subgroup, when that project includes a container registry with lots of container images. GitLab does not support moving projects with container images.

Prerequisites

• Maintainer privileges on SOURCE_PROJECT and TARGET_PROJECT

General Idea

SOURCE_PROJECT and TARGET_PROJECT are the same project, but before and after the move (at different URLs). TEMP_PROJECT is a separate project which only holds the container images for a while.

1. Copy all container images from SOURCE_PROJECT to a TEMP_PROJECT
2. Delete container images from SOURCE_PROJECT
3. Transfer SOURCE_PROJECT to TARGET_PROJECT via "Transfer Project" button
4. Copy all container images from TEMP_PROJECT back to TARGET_PROJECT
5. Remove TEMP_PROJECT

This procedure has the advantage that all network traffic is within GitLab CI, and the downlink to your local machine is not needed.

Detailed Instructions

1. Create temporary project TEMP_PROJECT
2. Enable container registry on TEMP_PROJECT. Pretty much all other features of TEMP_PROJECT except "Repository" can be disabled.
3. Add variable TEMP_PWD to CI variables of SOURCE_PROJECT containing an access token that allows pushing to TEMP_PROJECT's container registry. When in doubt, just use a universal api token
4. Add the following job to SOURCE_PROJECT's .gitlab-ci.yml:

   tj:move:
     stage: build
     image: 'debian:bookworm'
     variables:
       TARGET_IMAGE: 'registry.example.com/path/to/tjmove'
     script:
       - apt-get update
       - apt-get install -y jq ca-certificates skopeo
       - 'skopeo login --username "${CI_REGISTRY_USER}" --password "${CI_REGISTRY_PASSWORD}" ${CI_REGISTRY}'
       - |
         for tag in $(skopeo list-tags docker://${CI_REGISTRY_IMAGE} | jq -r .Tags[]); do
           if [[ -z $(skopeo inspect docker://${TARGET_IMAGE}:$tag 2>&1 >/dev/null) ]]; then
             echo "Skipping ${CI_REGISTRY_IMAGE}:$tag (already present at target)."
           else
             echo "Copying ${CI_REGISTRY_IMAGE}:$tag ..."
             skopeo copy --dest-creds "oauth2:${TEMP_PWD}" --retry-times 3 docker://${CI_REGISTRY_IMAGE}:$tag docker://${TARGET_IMAGE}:$tag
           fi
         done
     rules:
       - if: $TRIGGERVAR == "true"

5. Cancel the pipeline that was started for this commit.
6. Run the master pipeline with variable TRIGGERVAR set to true. This will run the above job.
7. Cancel any other jobs that were started along with the above job.
8. Keep an eye on the job. If it fails with weird I/O errors, just restart it until it completes successfully.
9. Remove TEMP_PWD from CI variables of SOURCE_PROJECT.
10. Remove the container registry from SOURCE_PROJECT.
11. TADAA Move SOURCE_PROJECT to TARGET_PROJECT.
12. Create a new container registry in TARGET_PROJECT.
13. Add variable TEMP_PWD again, but this time to CI variables of TARGET_PROJECT.
14. Modify the move job added above in TARGET_PROJECT's .gitlab-ci.yml to look like this:

    tj:move:
      stage: build
      image: 'debian:bookworm'
      variables:
        SOURCE_IMAGE: 'registry.example.com/path/to/tjmove'
      script:
        - apt-get update
        - apt-get install -y jq ca-certificates skopeo
        - 'skopeo login --username "${CI_REGISTRY_USER}" --password "${CI_REGISTRY_PASSWORD}" ${CI_REGISTRY}'
        - |
          for tag in $(skopeo list-tags --creds "oauth2:${TEMP_PWD}" docker://${SOURCE_IMAGE} | jq -r .Tags[]); do
            if [[ -z $(skopeo inspect docker://${CI_REGISTRY_IMAGE}:$tag 2>&1 >/dev/null) ]]; then
              echo "Skipping ${SOURCE_IMAGE}:$tag (already present at target)."
            else
              echo "Copying ${SOURCE_IMAGE}:$tag ..."
              skopeo copy --src-creds "oauth2:${TEMP_PWD}" --retry-times 3 docker://${SOURCE_IMAGE}:$tag docker://${CI_REGISTRY_IMAGE}:$tag
            fi
          done
      rules:
        - if: $TRIGGERVAR == "true"

    Make sure that the stage (here, build) really exists. Will cause "yaml invalid" if it doesn't.
15. Cancel the pipeline that was started for this commit.
16. Run the master pipeline with variable TRIGGERVAR set to true. This will run the above job.
17. Cancel any other jobs that were started along with the above job.
18. Keep an eye on the job. If it fails with weird I/O errors, just restart it until it completes successfully.
19. Check that the container registry in TARGET_PROJECT contains all the right content.
20. Remove TEMP_PWD from CI variables of TARGET_PROJECT.
21. Remove TEMP_PROJECT

Done!

Thanks for the great gist!

Slightly adapted to our self hosted GitLab instance:

# ⚠️ This job will only copy all the tags of the TARGET_IMAGE to the SOURCE_IMAGE. Needs adapting for multiple images if required.
interim_export_images_to_docker_poc:
  stage: build
  image: quay.io/containers/skopeo:v1.16.1-immutable@sha256:d3d1929b572bf9324d29c74605468beff8af837f1a11bd64497343009a4a4498
  variables:
    SOURCE_IMAGE: "${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}"  # convention image name == repository name
    TARGET_IMAGE: "gitlab-registry.company.com/group/subgroup/docker_poc/${CI_PROJECT_NAME}"
  script:
    - cat "${CI_SERVER_TLS_CA_FILE}" | tr -d '\r' >> /etc/ssl/certs/ca-bundle.crt  # to work with self signed certificates
    - dnf install -y jq
    - skopeo login --username "${CI_REGISTRY_USER}" --password "${CI_REGISTRY_PASSWORD}" ${CI_REGISTRY}
    - skopeo list-tags --debug "docker://${SOURCE_IMAGE}"
    - |
      for tag in $(skopeo list-tags "docker://${SOURCE_IMAGE}" | jq -r .Tags[]); do
        if skopeo inspect docker://${TARGET_IMAGE}:$tag ; then
          echo "Skipping ${SOURCE_IMAGE}:$tag (already present at target)."
        else
          echo "Copying ${CI_REGISTRY_IMAGE}:$tag ..."
          skopeo copy --dest-creds "oauth2:${MOVE_IMAGES_TOKEN}" --retry-times 3 docker://${SOURCE_IMAGE}:$tag docker://${TARGET_IMAGE}:$tag
        fi
      done
  rules:
    - if: $TRIGGER_COPY_OF_CONTAINER_IMAGES == "true"
    - when: manual

IIRC the else branch still does not trigger even if images exist, but skopeo worked well and fast every time so who cares. :D