Proxmox High Available cluster with Ceph and Dynamic Routing - Managing and Troubleshooting

Part of collection: Hyper-converged Homelab with Proxmox

This is part 3 focussing on Managing and Troubleshooting Proxmox and Ceph.

See also Part 1 about Setup Networking for a High Available cluster with Ceph, and see Part 2 for how to setup the Proxmox and Ceph Cluster itself.

WIP

My experiance Managing and Troubleshooting Proxmox and Ceph will go here!

Ceph won't Reinstal

Because I initially linked the Ceph Storage network to my Network Segment, I had to remove it myself. For this, I used this [How to Completely remove/delete or reinstall ceph and its configuration from Proxmox VE (PVE)](How to Completely remove/delete or reinstall ceph and its configuration from Proxmox VE (PVE)) guide. However, I encountered this error message: command 'cp /etc/pve/priv/ceph.client.admin.keyring /etc/ceph/ceph.client.admin.keyring' failed: exit code 1 (500). First, run pveceph createmon, and then proceed with the installation/configuration of Ceph again.

Remove OSD

This can be done through the GUI, but I was too quick and followed the guide Proxmox Ceph remove OSD – How to do it via Proxmox VE GUI and CLI?.

It wasn't clear to me what {name} should be in ceph osd crush remove {name}. After some searching here, it turned out to be ceph osd crush remove osd.x, where 'x' corresponds to the OSD number.

Module 'dashboard' has a failed dependency

PyO3 modules may only be initialized once per interpreter process," and it seems to be Ceph / Python related.

Destroy Ceph from Disk

Try this often works:

dd if=/dev/zero of=/dev/diskname bs=1M count=1000
fdisk /dev/diskname
Option G / Option N / enter enter enter / W

Remove Cephfs

Unmount /mnt/cephfs
Destroy MDS

Remove CephFS with the command ceph fs rm cephfs --yes-i-really-mean-it