Delete Stuck Namespace

README.md

Delete Stuck Namespace

Here be dragons. Use with extreme caution.

About

This script is designed to accept namespaces as a list of arguments and will perform a namespace child resource delete loop before terminating the namespace and any finalizers.

Motivation

Having Google'd the classic StackOverflow Deleted Namespace Stuck In Pending question for the five thousandth time I decided to make this a script in a gist.

bye.sh

#!/bin/bash

# Delete all resources in namespace
delete_resources_in_namespace() {
  local namespace=$1
  echo "Deleting all resources in namespace: $namespace"
  kubectl api-resources --verbs=list --namespaced -o name \
    | xargs -n 1 kubectl delete --all -n "$namespace" --ignore-not-found --wait
}

# Remove finalizers and delete stuck namespace
delete_namespace() {
  local namespace=$1
  echo "Removing finalizers and deleting namespace: $namespace"
  kubectl get namespace "$namespace" -o json \
    | jq 'del(.spec.finalizers)' \
    | kubectl replace --raw "/api/v1/namespaces/$namespace/finalize" -f -
  kubectl delete namespace "$namespace"
  echo "Waiting for namespace $namespace to be deleted..."
  while kubectl get namespace "$namespace" &>/dev/null; do
    sleep 1
  done
  echo "Namespace $namespace has been deleted."
}

# Check for at least one namespace provided as argument
if [ $# -eq 0 ]; then
  echo "Usage: $0 <namespace1> <namespace2> ... <namespaceN>"
  exit 1
fi

# Loop through and delete list of namespaces
for namespace in "$@"; do
  delete_resources_in_namespace "$namespace"
  delete_namespace "$namespace"
done

Excellent script Kat!

One addition/comment/suggestion

Even without finalizers, things may hang up due to failed volume mounts (for example due to a misconfigured CSI driver).

When that happens I end up doing something like:

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n $NS_TO_BE_NUKED

and then check if there is anything funny like FailedMount, or Pending, or things like that.

Especially failed CSI volume mounts are nasty because they prevent the pods from being terminated.

When that happens I

1. k delete deployment $the_deployment_of_the_pending_pod
2. k delete po $the_pod -n $the_ns --grace-period=0 --force

I’m not sure the best way to incorporate those to this script though.

I’ll play with it when I find some time.

Cheers,

V.

Ooh, thank you for the tip @v0lkan if I run into it I'll make a point to incorporate this technique into the script as well. Awesome stuff!