Namespace stuck in terminating state

namespace.md

Initial steps:

• Make sure that there are no resources within the namespace - Pods, Deployments, ReplicaSets, CRDs webhooks etc.
• Also, run kubectl get ns <your_namespace> -o json > stuck_ns.json and check the JSON to see if there are obvious indications as to what's causing it.

Approach 1 .

Right way is to find out why it's stuck in terminating state. Very common reason is there's an unavailable API service(s) which prevents cluster from finalizing namespaces.

1. kubectl get apiservice | grep False

2. If the grep was able to find a match, run the following command to check which resource was failing: kubectl get apiservice -o yaml

eg:

- apiVersion: apiregistration.k8s.io/v1
  kind: APIService
  metadata:
    annotations:
      ...
    creationTimestamp: ...
    labels:
      ...
    name: v1beta1.external.metrics.k8s.io
    ...
  spec:
    ...
  status:
    conditions:
    - lastTransitionTime: ...
      message: 'failing or missing response from https://10.30.69.78:8443/apis/external.metrics.k8s.io/v1beta1:
        Get https://10.30.69.78:8443/apis/external.metrics.k8s.io/v1beta1: dial tcp
        10.30.69.78:8443: connect: connection refused'
      reason: FailedDiscoveryCheck
      status: "False"
      type: Available

3. kubectl delete apiservice <api_resource> eg: kubectl delete apiservice v1beta1.external.metrics.k8s.io

Approach 2 - Bash script based on this article

#!/bin/bash
k8s_delete_ns=$1
echo "Provided namesapce: ${k8s_delete_ns}..."
echo "Exporting namespace configuration..."
kubectl get namespaces -o json | grep "${k8s_delete_ns}"
kubectl get namespace ${k8s_delete_ns} -o json > temp.json
echo "Opening editor..."
wait 3
vi temp.json
echo "Sending configuration to k8s master for processing..."
curl -H "Content-Type: application/json" -X PUT --data-binary @temp.json http://127.0.0.1:8080/api/v1/namespaces/${k8s_delete_ns}/finalize
echo "Waiting for namespace deletion to process..."
wait 12
kubectl get namespaces
echo "...done."