Kubernetes: restore deleted manifests wuth auger from etcd backup

One day I accidentally deleted the argoprojects and applications for argocd.

I could have restarted 300 pipelines and generated these templates, but I chose a different path.

This morning I read an article Post-exploiting a compromised etcd – Full control over the cluster and its nodes Seems, it's nice case to try auger in action.

Install auger, I built from 6922d3a04e360a144f166f73e4056c34b3472750 commit for macos

git clone https://github.com/jpbetz/auger
cd auger
make release GOOS=darwin
# or if you don't wanna use docker
make build GOOS=darwin

Download you etcd backup, extract .db file from it.

FILE=<etcd-backup>.db

Explore all keys stored in backup, there will be many lines, so redirect output to file.

auger extract -f $FILE > keys.txt

Example of keys.txt

/registry/argoproj.io/applications/argocd-ns/my-app
/registry/apiextensions.k8s.io/customresourcedefinitions/sectypes.internal.linstor.linbit.com
/registry/acme.cert-manager.io/orders/istio-system/tls-my-company-rr2qt-611775170
...

So let's extract our long-awaited manifests

To extract manifests from .db file, use this syntax

auger extract -f $FILE -k <key>

Example

$ auger extract -f $FILE -k /registry/namespaces/kube-system
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2021-02-09T12:09:05Z"
  name: kube-system
  uid: 8759e8a8-4154-4a70-9538-7585c70db458
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

But, if you try to extract custom resource, you will get an error

$ auger extract -f $FILE -k /registry/argoproj.io/applications/argocd-ns/my-app
...
Error: error decoding from application/json: no kind "Application" is registered for version "argoproj.io/v1alpha1"

AFAIK this means, that auger cannot find protobuf schema to validate manifest.

So you can just add --raw and get json output

$ auger extract -f $FILE -k /registry/argoproj.io/applications/argocd-ns/my-app --raw

Create a simple bash script to extract all our applications and deploy them. During the extraction scripts deletes .metadata.uid and .status keys, otherwise kubectl will throw error.

mkdir -p applications
for key in $(auger extract -f $FILE | grep -i "argoproj.io/applications"); do
  auger extract -f $FILE --raw -k "$key" | yq -P 'del(.status) | del(.metadata.uid) | del(.metadata.creationTimestamp)' > applications/$(basename $key.yaml)
done

Script was made quick-and-dirty, so any suggestions are welcome.

Finally apply all this manifests

kubectl apply -f applications

paraddise/Kubernetes: restore deleted manifests wuth auger from etcd backup.md