Skip to content

Instantly share code, notes, and snippets.

@chrishoerl
Last active September 13, 2024 13:00
Show Gist options
  • Save chrishoerl/3c33609aa451a5a228d53a5f24a2def2 to your computer and use it in GitHub Desktop.
Save chrishoerl/3c33609aa451a5a228d53a5f24a2def2 to your computer and use it in GitHub Desktop.
Bulk delete gitlab pipelines older than a given date
#!/bin/bash
# Purpose: Bulk-delete GitLab pipelines older than a given date
# Author: github.com/chrishoerl
# GitLab API: v4
# Requirements: jq must be instaled ($ sudo apt install jq)
# API example: https://gitlab.example.com/api/v4/projects
# API example: https://gitlab.example.com/api/v4/projects/<projectid>/pipelines
#
# NOTE: Script is just a dryrun. To really delete pipelines, simply uncomment line 49 to activate
#
################### FIRST REPLACE VARIABLES WITH YOR VALUES ############
# Define some variables
GITLABURL="https://gitlab.example.com"
ACCESSTOKEN="XXXXXXXXXXXXX" #must have API r/w access
# Pipelines older than this date will be deleted by this script
DELETEBEFORE="2020-01-16" #date range format: yyyy-mm-dd
########################################################################
# Get all project names and write to array
project_array=( $(curl --header "PRIVATE-TOKEN: $ACCESSTOKEN" "$GITLABURL/api/v4/projects?per_page=100" 2> /dev/null | jq -r '.[] | .path_with_namespace') )
# Now let's work with the project names from the array
COUNTER=1
for PROJECTNAME in "${project_array[@]}"
do
# debug
#echo "Project: $PROJECTNAME"
# Determine project id of given project name
PROJECTID=`curl --header "PRIVATE-TOKEN: $ACCESSTOKEN" $GITLABURL/api/v4/projects?per_page=100 2> /dev/null | jq '.[]|select(.path_with_namespace=="'$PROJECTNAME'")|.id'`
# Find out pipeline IDs of our project which match the date range and write results into array
pipeline_array=( $(curl --header "PRIVATE-TOKEN: $ACCESSTOKEN" "$GITLABURL/api/v4/projects/$PROJECTID/pipelines?per_page=100&updated_before="$DELETEBEFORE"T23:01:00.000Z" 2> /dev/null | jq -r '.[] | .id') )
# Count how many pipelines we found out
NUMOFPIPES=`echo ${#pipeline_array[@]}`
# Print statistics
echo -e "---\nTask $COUNTER: Project $PROJECTNAME (ID:$PROJECTID) has $NUMOFPIPES pipelines to delete."
# Print all pipeline IDs from the array
for pipelineid in "${pipeline_array[@]}"
do
echo "Deleting pipeline ID: ${pipelineid}"
echo "$GITLABURL/api/v4/projects/$PROJECTID/pipelines/${pipelineid}"
## ACTIVATE THIS TO START CLEANUP JOB
## Create DELETE query for each pipeline matching our date range
#curl --header "PRIVATE-TOKEN: $ACCESSTOKEN" --request "DELETE" "$GITLABURL/api/v4/projects/$PROJECTID/pipelines/$pipelineid"
done
# Raise counter
COUNTER=$((COUNTER + 1))
done
@xosofox
Copy link

xosofox commented Jul 13, 2021

Dynamic delete date with:

line 16
DELETEBEFORE=$(/bin/date --date="45 days ago" +%Y-%m-%d) #date range format: yyyy-mm-dd

@anutator
Copy link

anutator commented Oct 8, 2022

Your script doesn't work when number of projects is more than 100. Besides I found some troubles with archived projects. And is't easier to have array from project id numbers and find project name by id, I changed this also because of some problems after updating to Gitlab 14.10.5.
There is a new way to get a list of all projects in Giltab 15.x, but I didn't test it yet? I'll test after upgrade.
Your script also doesn't delete pipelines more than 100, so I add a while cycle to get it work. Here is my script. Please test and check If you can make it clear.
https://gist.github.com/anutator/7f24f83db93bf7cc5d159de4fabd77a7

@LeeRoyManea
Copy link

LeeRoyManea commented Nov 10, 2023

#!/bin/bash
# Purpose: Bulk-delete GitLab pipelines older than a given date
# Author: github.com/chrishoerl  Updated github.com/LeeRoyManea
# GitLab API: v4
# Requirements: jq must be instaled ($ sudo apt install jq)
# API example: https://gitlab.example.com/api/v4/projects
# API example: https://gitlab.example.com/api/v4/projects/<projectid>/pipelines
#
# NOTE: Script is just a dryrun. To really delete pipelines, simply uncomment line 49 to activate
################### FIRST REPLACE VARIABLES WITH YOUR VALUES ############
# Define some variables
GITLABURL=""
ACCESSTOKEN="" # must have API r/w access
########################################################################

# Function to get all items from a paginated GitLab API endpoint
get_all_items() {
  local endpoint=$1
  local array_name=$2
  local page=1
  while true; do
    response=$(curl --header "PRIVATE-TOKEN: $ACCESSTOKEN" "$GITLABURL/$endpoint?page=$page&per_page=100" 2>/dev/null)
    #current_items=$(echo "$response" | jq -r '.[]? | .id // empty')
    current_items=$(echo "$response" | jq -r 'if length > 0 then .[] | .id else empty end')
    if [ -z "$current_items" ]; then
      break
    fi
    eval "$array_name+=( $current_items )"
    page=$((page + 1))
    #echo "loading Page: ${page} of '${GITLABURL}/${endpoint}'"
  done
}

# Get all project IDs and write to array
project_ids=()
get_all_items "api/v4/projects" project_ids

# Now let's work with the project IDs from the array
COUNTER=1
for PROJECTID in "${project_ids[@]}"; do
  # Get project name
  PROJECTNAME=$(curl --header "PRIVATE-TOKEN: $ACCESSTOKEN" "$GITLABURL/api/v4/projects/$PROJECTID" 2>/dev/null | jq -r '.path_with_namespace')

  # Find out pipeline IDs of our project which match the date range and write results into array
  pipeline_array=()
  get_all_items "api/v4/projects/$PROJECTID/pipelines" pipeline_array

  # Count how many pipelines we found out
  NUMOFPIPES=${#pipeline_array[@]}

  # Print statistics
  echo -e "---\nTask $COUNTER: Project $PROJECTNAME (ID:$PROJECTID) has $NUMOFPIPES pipelines to delete."

  # Print all pipeline IDs from the array
  for pipelineid in "${pipeline_array[@]}"; do
    echo "Deleting pipeline ID: ${pipelineid} -> $GITLABURL/api/v4/projects/$PROJECTID/pipelines/${pipelineid}"
    curl --header "PRIVATE-TOKEN: $ACCESSTOKEN" --request "DELETE" "$GITLABURL/api/v4/projects/$PROJECTID/pipelines/$pipelineid"
  done
  # Raise counter
  COUNTER=$((COUNTER + 1))
done

this is working for me to delete all pipelines in all projects. not only the top 100. sadly the datetime filter causing issues for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment