This gist is a follow-up to the CSI Integration blog post on the Velero website.

This gist is a follow-up to the CSI Integration blog post on the Velero website. Updates here include:

Using v0.1.2 release of velero-plugin-for-csi
Using [v1.1.1 release of velero-plugin-for-microsoft-azure](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/releases/tag/v1.1.1)
Using v1.5.2 release of Velero

Below are the updated set of instructions to

Create AKS cluster with kubernetes version v1.17 or later. In this example an AKS cluster running v1.18.6 is used.

Install CSI driver

curl -skSL https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/install-driver.sh | bash -s master snapshot --

Install Velero

VELERO_IMAGE=velero/velero:v1.5.2
VELERO_PLUGINS=velero/velero-plugin-for-microsoft-azure:v1.1.1,velero/velero-plugin-for-csi:v0.1.2
source $HOME/bin/velero-dev/aks-env
velero install \
--provider azure \
--plugins ${VELERO_PLUGINS} \
--bucket $BLOB_CONTAINER \
--secret-file aks-creds \
--backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID \
--snapshot-location-config apiTimeout=5m,resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID \
--image ${VELERO_IMAGE} \
--features=EnableCSI

Verify installed Velero version and the version of the CLI

$ velero version
Client:
    Version: v1.5.2
    Git commit: e115e5a191b1fdb5d379b62a35916115e77124a4
Server:
    Version: v1.5.2

Verify installation of velero-plugin-for-microsfot-azure and velero-plugin-for-csi

$ velero plugin get | grep csi
velero.io/csi-pvc-backupper                     BackupItemAction
velero.io/csi-volumesnapshot-backupper          BackupItemAction
velero.io/csi-volumesnapshotclass-backupper     BackupItemAction
velero.io/csi-volumesnapshotcontent-backupper   BackupItemAction
velero.io/csi-pvc-restorer                      RestoreItemAction
velero.io/csi-volumesnapshot-restorer           RestoreItemAction
velero.io/csi-volumesnapshotclass-restorer      RestoreItemAction
velero.io/csi-volumesnapshotcontent-restorer    RestoreItemAction
$ velero plugin get | grep azure
velero.io/azure                                 ObjectStore
velero.io/azure                                 VolumeSnapshotter

Create storage class and volumesnapshot class

apiVersion: storage.k8s.io/v1 
kind: StorageClass 
metadata: 
    name: disk.csi.azure.com 
provisioner: disk.csi.azure.com 
parameters: 
    skuname: StandardSSD_LRS 
reclaimPolicy: Delete 
volumeBindingMode: Immediate 
allowVolumeExpansion: true
---
apiVersion: snapshot.storage.k8s.io/v1beta1 
kind: VolumeSnapshotClass 
metadata: 
    name: csi-azuredisk-vsc
    labels:
        velero.io/csi-volumesnapshot-class: "true"
driver: disk.csi.azure.com 
deletionPolicy: Retain 
parameters:
    tags: 'foo=aaa,bar=bbb'

From StorageClass and VolumeSnapshotClass. NOTE: The volumesnapshotclass was edited to add the velero.io/csi-volumesnapshot-class: "true" annotation.

Deploy a sample CSI app

apiVersion: v1
kind: Namespace
metadata:
creationTimestamp: null
name: csi-app
---
kind: Pod
apiVersion: v1
metadata:
namespace: csi-app
name: csi-nginx
spec:
nodeSelector:
    kubernetes.io/os: linux
containers:
    - image: nginx
    name: nginx
    command: [ "sleep", "1000000" ]
    volumeMounts:
        - name: azuredisk01
        mountPath: "/mnt/azuredisk"
volumes:
    - name: azuredisk01
    persistentVolumeClaim:
        claimName: pvc-azuredisk
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: csi-app
name: pvc-azuredisk
spec:
accessModes:
    - ReadWriteOnce
resources:
    requests:
    storage: 1Gi
storageClassName: disk.csi.azure.com
---

Make our sample app generate some data into the PV

$ kubectl -n csi-app exec -ti csi-nginx -- bash
root@csi-nginx:/# fallocate -l 750M /mnt/azuredisk/velero-data
root@csi-nginx:/# cksum /mnt/azuredisk/velero-data
3090901991 786432000 /mnt/azuredisk/velero-data
root@csi-nginx:/#

Take a velero backup

$ velero backup create csi-b1 --include-namespaces csi-app --wait
Backup request "csi-b1" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
.......
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe csi-b1` and `velero backup logs csi-b1`.

Describe the backup to confirm that the CSI volumesnapshots were included in the backup

$ velero backup describe csi-b1 --details --features=EnableCSI
Name:         csi-b1
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.18.6
            velero.io/source-cluster-k8s-major-version=1
            velero.io/source-cluster-k8s-minor-version=18

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
Included:  csi-app
Excluded:  <none>

Resources:
Included:        *
Excluded:        <none>
Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2020-10-29 15:20:57 -0700 PDT
Completed:  2020-10-29 15:21:03 -0700 PDT

Expiration:  2020-11-28 14:20:57 -0800 PST

Total items to be backed up:  20
Items backed up:              20

Resource List:
snapshot.storage.k8s.io/v1beta1/VolumeSnapshot:
    - csi-app/velero-pvc-azuredisk-7dbp7
snapshot.storage.k8s.io/v1beta1/VolumeSnapshotClass:
    - csi-azuredisk-vsc
snapshot.storage.k8s.io/v1beta1/VolumeSnapshotContent:
    - snapcontent-2934eb73-6dd3-494a-91ef-7e52ad8115b9
v1/Event:
    - csi-app/csi-nginx.1642958b3a639423
    - csi-app/csi-nginx.1642958bd742db20
    - csi-app/csi-nginx.1642958df050f87b
    - csi-app/csi-nginx.16429592eb598457
    - csi-app/csi-nginx.1642959651292777
    - csi-app/csi-nginx.164295970bedfe18
    - csi-app/csi-nginx.16429597e182a29f
    - csi-app/csi-nginx.16429597eae448f1
    - csi-app/pvc-azuredisk.1642958b4da4a67a
    - csi-app/pvc-azuredisk.1642958b4dce84d8
    - csi-app/pvc-azuredisk.1642958bf941e107
v1/Namespace:
    - csi-app
v1/PersistentVolume:
    - pvc-ba8cc2e9-8255-4bda-bace-013f1036d693
v1/PersistentVolumeClaim:
    - csi-app/pvc-azuredisk
v1/Pod:
    - csi-app/csi-nginx
v1/Secret:
    - csi-app/default-token-vsh5n
v1/ServiceAccount:
    - csi-app/default

Velero-Native Snapshots: <none included>

CSI Volume Snapshots:
Snapshot Content Name: snapcontent-2934eb73-6dd3-494a-91ef-7e52ad8115b9
Storage Snapshot ID: /subscriptions/489d6686-11b0-4662-a8e5-4143cd7cd6af/resourceGroups/mc_ashisha-aks_velero-dev_eastus/providers/Microsoft.Compute/snapshots/snapshot-2934eb73-6dd3-494a-91ef-7e52ad8115b9
Snapshot Size (bytes): 1073741824
Ready to use: true

Simulate a disaster, Delete the csi-app namespace

$ kubectl delete ns csi-app
namespace "csi-app" deleted
$ kubectl get ns csi-app
Error from server (NotFound): namespaces "csi-app" not found

Restore from the backup taken in above

$ velero create restore --from-backup csi-b1 --wait
Restore request "csi-b1-20201029153424" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
.
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe csi-b1-20201029153424` and `velero restore logs csi-b1-20201029153424`.

Verify that the csi-app was restored completely with the data intact

$ kubectl get ns csi-app
NAME      STATUS   AGE
csi-app   Active   81s
$ kubectl -n csi-app exec -ti csi-nginx -- bash
root@csi-nginx:/# ls /mnt/azuredisk/
lost+found/  velero-data
root@csi-nginx:/# ls /mnt/azuredisk/velero-data
/mnt/azuredisk/velero-data
root@csi-nginx:/# cksum /mnt/azuredisk/velero-data
3090901991 786432000 /mnt/azuredisk/velero-data
root@csi-nginx:/#

ashish-amarnath/velero-csi-plugin-demo.md

viktor1298-dev commented Dec 8, 2021