Last active
November 2, 2020 19:16
-
-
Save TJM/31b68f12b2bc962141ce1eb27f7bcb10 to your computer and use it in GitHub Desktop.
Dealing with Orphaned pod messages (Orphaned pod found - but volume paths are still present on disk)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash -eu | |
# | |
# This script is designed to be run as a cron job periodically to | |
# clean up the Orphaned Pods. Use at your own risk! | |
# | |
## Settings (can be passed as environment variables) | |
LOGFILE=${LOGFILE:-/var/log/messages} # what log file to process | |
KUBELET_PODS_DIR=${KUBELET_DIR:-/var/lib/kubelet/pods} # where to find pods to remove | |
DEBUG=${DEBUG:-0} # more debug output | |
## Script Starts Here | |
last_pod="" | |
# Retrieve the most recent "Orphaned pod" log entry (tac = backwards cat) | |
# NOTE: If there are no orphaned pods, then pod==last_pod, so its skips :) | |
while pod=$(tac ${LOGFILE} | grep -m 1 'Orphaned pod' | grep -o '".*"' | sed 's/"//g'); do | |
if [[ $pod != $last_pod ]]; then | |
last_pod=$pod | |
pod_dir="${KUBELET_PODS_DIR}/${pod}" | |
if [[ -d $pod_dir ]]; then | |
# echo "$pod exist" | |
rm -rf "${pod_dir}/volumes" | |
if [[ $? == 0 ]]; then | |
echo "Cleaned orphaned pod volumes: $pod" | |
else | |
echo "ERROR: Pod cannot be deleted: ${pod_dir}" >&2 | |
fi | |
else | |
[[ $DEBUG == 1 ]] && echo "DEBUG: $pod doesn't exist in directory (${pod_dir})" | |
fi | |
else | |
[[ $DEBUG == 1 ]] && echo "DEBUG: No (more) orphaned pods found." | |
break | |
fi | |
sleep 1 | |
done |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I would like to see if we can make the "find most recent Orphaned pod" more efficient:
I feel like there is probably a better way to handle that... I am also not sure whether using
tac LOGFILE | grep -m 1 'match'
is more or less efficient thangrep 'match' LOGFILE | tail -1