Using pv-migrate, prometheus monitoring data can be migrated between PV/PVCs when migrating to monitoring v2
This assumes persistent storage is used with monitoring (ie, a PV/PVC exists) and is intended only for cluster monitoring (not project monitoring).
- Monitoring v1 apps (in Cluster Manager) should be disabled (Tools > Monitoring)
- Ensure the monitoring v1 apps are uninstalled, more details here
- Monitoring v2 (in Cluster Explorer) should be installed
- Install pv-migrate, steps available here
- Configure a kubeconfig for the cluster
- Check the source and destination PV/PVCs exist:
kubectl get pv,pvc -n cattle-monitoring-system
kubectl get pv,pvc -n cattle-prometheus
- Scale down the prometheus statefulset, no pod should be running during the process:
kubectl patch prometheus -n cattle-monitoring-system rancher-monitoring-prometheus --type='merge' -p '{"spec":{"replicas":0}}'
- Migrate the data:
pv-migrate migrate -i --source-namespace cattle-prometheus --dest-namespace cattle-monitoring-system --dest-delete-extraneous-files --dest-path /prometheus-db prometheus-cluster-monitoring-db-prometheus-cluster-monitoring-0 prometheus-rancher-monitoring-prometheus-db-prometheus-rancher-monitoring-prometheus-0
- Scale prometheus back up:
kubectl patch prometheus -n cattle-monitoring-system rancher-monitoring-prometheus --type='merge' -p '{"spec":{"replicas":1}}'
Once started, the prometheus tsdb data should be available, confirm when viewing in Grafana
Remove the cattle-prometheus namespace, this should clean up any remaining objects for monitoring v1, including deleting the PV/PVC (note: the PV disk will be deleted, data will be deleted also).
kubectl delete namespace cattle-prometheus