datavolume-disaster-sim.md

KubeVirt DataVolumes are Disaster Recovery

KubeVirt DataVolumes are currently not safe to use in a gitops DR flow. Here's why.

VM Definition

cat <<EOF | k create -f -
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: testvm
  name: testvm
spec:
  dataVolumeTemplates:
  - metadata:
      name: test-dv1
    spec:
      pvc:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 500Mi
        storageClassName: rook-ceph-block
      source:
        http:
          url: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/vm: testvm
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: datavolumedisk1
          - disk:
              bus: virtio
            name: cloudinitdisk
        resources:
          requests:
            memory: 64M
      volumes:
      - dataVolume:
          name: test-dv1
        name: datavolumedisk1
      - cloudInitNoCloud:
          userData: |
            #!/bin/sh

            echo 'printed from cloud-init userdata'
        name: cloudinitdisk
EOF

Make VM Special

virtctl console testvm
  ____               ____  ____
 / __/ __ ____ ____ / __ \/ __/
/ /__ / // __// __// /_/ /\ \
\___//_//_/  /_/   \____/___/
   http://cirros-cloud.net


login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
testvm login: cirros
Password:
$
$ echo "Hello World" > message.txt
$ cat message.txt

Stop the VM

virtctl stop testvm

Simulate Pre Failover Environment

PV_NAME=$(k get pvc test-dv1 -o=jsonpath='{.spec.volumeName}')
k patch pv $PV_NAME -p '{"spec": {"persistentVolumeReclaimPolicy": "Retain"}}'
k delete vm testvm
k patch pv $PV_NAME -p '{"spec": {"claimRef": {"resourceVersion": "", "uid": ""}}}'
k patch pv $PV_NAME --type=json -p "[{'op': 'remove', 'path': '/metadata/annotations'}]"

Observe PV

See that the PV is Available and spec.claimRef is set for default/test-dv1

k get pv $PV_NAME -o yaml

kind: PersistentVolume
...
spec:
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: test-dv1
    namespace: default
...
status:
  phase: Available

Simulate Failover

Create the VM again from above

virtctl console testvm

  ____               ____  ____
 / __/ __ ____ ____ / __ \/ __/
/ /__ / // __// __// /_/ /\ \
\___//_//_/  /_/   \____/___/
   http://cirros-cloud.net


login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
testvm login: cirros
Password:
$ ls message.txt
ls: message.txt: No such file or directory