Gitlab: Clone / Pull all projects in a group

sync-projects

#!/usr/bin/env bash

# Documentation
# https://docs.gitlab.com/ce/api/projects.html#list-projects

NAMESPACE="YOUR_NAMESPACE"
BASE_PATH="https://gitlab.example.com/"
PROJECT_SEARCH_PARAM=""
PROJECT_SELECTION="select(.namespace.name == \"$NAMESPACE\")"
PROJECT_PROJECTION="{ "path": .path, "git": .ssh_url_to_repo }"

if [ -z "$GITLAB_PRIVATE_TOKEN" ]; then
    echo "Please set the environment variable GITLAB_PRIVATE_TOKEN"
    echo "See ${BASE_PATH}profile/account"
    exit 1
fi

FILENAME="repos.json"

trap "{ rm -f $FILENAME; }" EXIT

curl -s "${BASE_PATH}api/v3/projects?private_token=$GITLAB_PRIVATE_TOKEN&search=$PROJECT_SEARCH_PARAM&per_page=999" \
    | jq --raw-output --compact-output ".[] | $PROJECT_SELECTION | $PROJECT_PROJECTION" > "$FILENAME"

while read repo; do
    THEPATH=$(echo "$repo" | jq -r ".path")
    GIT=$(echo "$repo" | jq -r ".git")
    
    if [ ! -d "$THEPATH" ]; then
        echo "Cloning $THEPATH ( $GIT )"
        git clone "$GIT" --quiet &
    else
        echo "Pulling $THEPATH"
        (cd "$THEPATH" && git pull --quiet) &
    fi
done < "$FILENAME"

wait

I simplify it to this script:

#!/usr/bin/env bash

BASE_PATH="http://10.21.6.54:6060/"

# GITLAB_PRIVATE_TOKEN=Rq6EnxsQyepvUyH8Dv2J

if [ -z "$1" ]
  then
    echo "group name is required."
    exit 1;
fi

GROUP_NAME="$1"

if [ -z "$GITLAB_PRIVATE_TOKEN" ]; then
    echo "Please set the environment variable GITLAB_PRIVATE_TOKEN"
    echo "See ${BASE_PATH}profile/account"
    exit 1
fi

FIELD_NAME="ssh_url_to_repo"

echo "Cloning all git projects in group $GROUP_NAME";

REPO_SSH_URLS=`curl -s "${BASE_PATH}api/v3/groups/$GROUP_NAME/projects?private_token=$GITLAB_PRIVATE_TOKEN&per_page=999" \
   | grep -o "\"$FIELD_NAME\":[^ ,]\+" | awk -F'"' '{print $4}' | grep $GROUP_NAME`

for REPO_SSH_URL in $REPO_SSH_URLS; do
    THEPATH=$(echo "$REPO_SSH_URL" | awk -F'/' '{print $NF}' | awk -F'.' '{print $1}')

    if [ ! -d "$THEPATH" ]; then
        echo "Cloning $THEPATH ( $REPO_SSH_URL )"
        git clone "$REPO_SSH_URL" --quiet &
    else
        echo "Pulling $THEPATH"
        (cd "$THEPATH" && git pull --quiet) &
    fi
done

I simplify it to this script:

REPO_SSH_URLS=`curl -s "${BASE_PATH}api/v3/groups/$GROUP_NAME/projects?private_token=$GITLAB_PRIVATE_TOKEN&per_page=999" \

For GitLab 11.8, v3 API is deprecated and no longer works. Change v3 to v4 in the URL to get it working.

I simplify it to this script:

#!/usr/bin/env bash

BASE_PATH="http://10.21.6.54:6060/"

# GITLAB_PRIVATE_TOKEN=Rq6EnxsQyepvUyH8Dv2J

if [ -z "$1" ]
  then
    echo "group name is required."
    exit 1;
fi

GROUP_NAME="$1"

if [ -z "$GITLAB_PRIVATE_TOKEN" ]; then
    echo "Please set the environment variable GITLAB_PRIVATE_TOKEN"
    echo "See ${BASE_PATH}profile/account"
    exit 1
fi

FIELD_NAME="ssh_url_to_repo"

echo "Cloning all git projects in group $GROUP_NAME";

REPO_SSH_URLS=`curl -s "${BASE_PATH}api/v3/groups/$GROUP_NAME/projects?private_token=$GITLAB_PRIVATE_TOKEN&per_page=999" \
   | grep -o "\"$FIELD_NAME\":[^ ,]\+" | awk -F'"' '{print $4}' | grep $GROUP_NAME`

for REPO_SSH_URL in $REPO_SSH_URLS; do
    THEPATH=$(echo "$REPO_SSH_URL" | awk -F'/' '{print $NF}' | awk -F'.' '{print $1}')

    if [ ! -d "$THEPATH" ]; then
        echo "Cloning $THEPATH ( $REPO_SSH_URL )"
        git clone "$REPO_SSH_URL" --quiet &
    else
        echo "Pulling $THEPATH"
        (cd "$THEPATH" && git pull --quiet) &
    fi
done

Thank you, all of you, for helping me with this. This is what I have so far.

#!/usr/bin/env bash

BASE_PATH="http://192.168.1.156/"

GITLAB_PRIVATE_TOKEN=iZardfn3AWpxsRM1zjzP

if [ -z "$GITLAB_PRIVATE_TOKEN" ]; then
echo "Please set the environment variable for GITLAB_PRIVATE_TOKEN"
echo "See ${BASE_PATH}profile/account"
exit 1
fi

FIELD_NAME="ssh_url_to_repo"

echo "Cloning or pulling updates on all GitLab projects."

REPO_SSH_URLS=curl -s "${BASE_PATH}api/v4/projects?private_token=$GITLAB_PRIVATE_TOKEN&per_page=999" \ | grep -o "\"$FIELD_NAME\":[^ ,]\+" | awk -F'"' '{print $4}'

echo ""
echo $REPO_SSH_URLS
echo ""

for REPO_SSH_URL in $REPO_SSH_URLS; do
echo $REPO_SSH_URL

REPO_DIR=$(echo "$REPO_SSH_URL" | awk -F'/' '{print $NF}' | awk -F'.' '{print $1}')

if [ ! -d "$REPO_DIR" ]; then
    echo "Cloning $REPO_DIR ( $REPO_SSH_URL )"
    git clone "$REPO_SSH_URL" --quiet &
else
    echo "Pulling $REPO_DIR"
    (cd "$REPO_DIR" && git pull --quiet) &
fi

done

How would I delete directories in the current working directory that the script is running that don't have projects in GitLab?

Lots of good solutions detailed above. However, you can also try using ghorg which is small cli that will do most of the work for you. It supports github, gitlab, and bitbucket.

Lots of good solutions detailed above. However, you can also try using ghorg which is small cli that will do most of the work for you. It supports github, gitlab, and bitbucket.

This doesn't seem to work with GitLab Community edition, only gitlab.com, right?

I opened an issue on ghorg and you can now specify your gitlab instance: gabrie30/ghorg#41

Also: https://github.com/angristan/gitlab-repo-dl

https://github.com/ezbz/gitlabber/

Lots of good solutions detailed above. However, you can also try using ghorg which is small cli that will do most of the work for you. It supports github, gitlab, and bitbucket.

This doesn't seem to work with GitLab Community edition, only gitlab.com, right?

No, it supports all gitlab installations. If you run into any problems feel free to raise issue.

https://github.com/ozaydinb/gitlab-clonner I was inspired by this gist. my version is to clone all projects and sub-projects to different directories. it works for me, without any issue.

Hey @JonasGroeger , when I tried using your script, got an error of "parse error: Invalid numeric literal at line 1, column 16". Not sure exactly what Iam giving wrong, here is the snippet of my code -

#!/usr/bin/env bash

Documentation

https://docs.gitlab.com/ce/api/projects.html#list-projects

NAMESPACE="fsgbu-pbsm/fsgbu-ofspacs/dimension-management"
GITLAB_PRIVATE_TOKEN="xxxx7fcxx32xxxxw4xx"
BASE_PATH="https://cloudlab.us.oracle.com/"
PROJECT_SEARCH_PARAM=""
PROJECT_SELECTION="select(.namespace.name == "$NAMESPACE")"
PROJECT_PROJECTION="{ "path": .path, "git": .ssh_url_to_repo }"

if [ -z "$GITLAB_PRIVATE_TOKEN" ]; then
echo "Please set the environment variable GITLAB_PRIVATE_TOKEN"
echo "See ${BASE_PATH}profile/account"
exit 1
fi

FILENAME="repos.json"
............

Note-> I need to pull all subgroups under group dimension-management.

I wanted to clone also subgroups while keeping the tree structure. Since none of the solutions worked flawless, I wrote my own simple script => https://github.com/adroste/gitlab-clone-group

这是来自QQ邮箱的假期自动回复邮件。您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。