Create Erasure Coded CephFS Pools.md

Create Erasure Coded CephFS Pools

Part of collection: Hyper-converged Homelab with Proxmox

How to create a Erasure Coded Pool in Ceph and use 'directory pinning' to connect it to the CephFS filesystem.

To use a Erasure Coded Pool with CephFS, a directory inside the CephFS filesystem needs to be connected to a Erasure Coded Pool, this is called 'directory pinning'.

---

Create a Directory

The directory needs to be created inside the CephFS filesystem folder on one of the Proxmox Hosts.

mkdir /mnt/pve/cephfs/test-pool

This only has to be done only on one of the Proxmox Ceph cluster nodes!

---

Create a Pool

Now create an Erasure Coded Ceph Pool, this example uses the Ceph Dashboard; but it can also be done via the command-line.

Open the Ceph Dashboard, if it's installed, it can be found at: http://192.168.1.11:8443

Create a Erasure code profile

Create the pool

Add an EC Profile

Fill in the details and click + next to 'Erasure code profile'.

Create EC Profile 1

Create EC Profile

Select the desired settings; this example stores the Data on large but slower SSD drives, avoiding storing it on the faster NVMe in the cluster.

Create EC Profile 2

---

Directory Pinning

Add the folder to the CepFS File system using 'directory pinning'.

Open the terminal on a Host:

Add the Data Pool to CephFS

ceph fs add_data_pool cephfs test-pool

Connect the Data Pool to the directory.

setfattr -n ceph.dir.layout -v pool=test-pool /mnt/pve/cephfs/test-pool/

---

Now the folder is ready to be used with Proxmox VM's via Virtio-fs or other methods.

Ahh that must be the ceph dashboard I have heard about, sounds like it has options proxmox doesn’t have in their ui. I will go read up….

Yes, it has options that Proxmox doesn’t have and otherwise require command line commands. Dashboard is nice and useful, also to check if data is stored on the correct pools and or disk type.

Don’t worry about the Ceph warning after installation,
Module 'restful' has failed dependency: PyO3 modules may only be initialized once per interpreter process
nothing is wrong, just a bug in an upstream project.

oh wow, I feel you helped me dodge a bullet, the idea of cephFS with replicated pool (now I understand the difference) makes me nervous - even if I am sure only on swarm service would write to any give file…. Do you find the perf impact to be an issue?

Performance is not yet battle tested, but ‘brute force’ testing looks very good, even from a Vm with virtiofs, I get almost identical performance as on the host.

Performance is not yet battle tested, but ‘brute force’ testing looks very good, even from a Vm with virtiofs, I get almost identical performance as on the host.

sweet, now I know what i was doing wrong importing my VMs from Hyper-V to proxmox (don't ask) and the thunderbolt-net mesh is now rock solid i plan to:

1. move over all 3 docker VMs with their two disks (not getting rid of gluster at this point)
2. create a test debian VM using the same instructions I used for the docker VMs
3. test virtiofs like using your recipe and also test one or more of those ceph docker volume drivers too ( i want to prove my thesis on that approach - but i think I like virtiofs more.

I am hoping out there somewhere there is docker container with benchmarking tools inside it... might take me a couple of weekends to do all of this depending on what else i have going on

I let it all brew over the weekend, before I make a ‘final’ design how to proceed with my Docker Swarm and Homelab / Media service’s.

I guess for disk benchmarking this one could work.

Ahh that must be the ceph dashboard I have heard about, sounds like it has options proxmox doesn’t have in their ui. I will go read up….

Yes, it has options that Proxmox doesn’t have and otherwise require command line commands. Dashboard is nice and useful, also to check if data is stored on the correct pools and or disk type.

Don’t worry about the Ceph warning after installation, Module 'restful' has failed dependency: PyO3 modules may only be initialized once per interpreter process nothing is wrong, just a bug in an upstream project.

Thanks good to know, i had held off installing it because of that, any reason not to install the dashboard in say a privileged debian 12 LXC? (to keep the host os clean)

any reason not to install the dashboard in say a privileged debian 12 LXC? (to keep the host os clean)

I think it needs to be on a host, activating CEPH DASHBOARD, because it’s a Ceph Module. ‘ceph mgr module enable dashboard’

I think it needs to be on a host, activating CEPH DASHBOARD, because it’s a Ceph Module. ‘ceph mgr module enable dashboard’

Got it. I see i need to install the dashboard on all nodes.

do i also do this stuff on all nodes?

ceph mgr module enable dashboard --force  <this is my add, won't enable without this.
echo "thepassword" > /root/password
ceph dashboard ac-user-create cephdash -i ./password administrator --force
ceph config-key set mgr/dashboard/server_addr 127.0.0.1
ceph dashboard create-self-signed-cert
ceph mgr module disable dashboard
ceph mgr module enable dashboard
systemctl restart ceph-mgr@[hostname].service
systemctl status [email protected]
rm password

--edit-- oh i see, just read the docs - dashboard has to be on all nodes, ok will do on all nodes

oh interesting i can't execute any of the ceph dashboard commands or any command like ceph config-key get mgr/dashboard/crt - i get EONT errors

dashboard seems borked on my install (PVE8 + reef) https://forum.proxmox.com/threads/ceph-warning-post-upgrade-to-v8.129371/post-591377

finally got the dashboard working, fixes went into promox packages ~Feb

@scyto Better late than never 👊🏻