CephFS Mounting for Docker VMs (first draft)

2025.04.27 - currently untested e2e i had chatgpt make this for me based on the process i used with chatgpt to help figure it out so E&OE ...

This document describes the clean, final method to mount a CephFS filesystem for Docker VMs across your cluster.

Assumtions:

you have a working cephFS volume called docker (out of scope)
that you can see this just fine mounted on all 3 pve nodes (if you can't then this is never going to work)
that you are using the IPv6 version of my ceph proxmox setup
it assume you have full connectivity from within the VM to the internet and the ceph network - this relies on my new routed mesh network setup i haven't yet published

🛠️ Proxmox Node Setup (one-time, performed on any node)

1. Create a restricted CephFS client

ceph auth get-or-create client.docker-cephfs \
  mon 'allow r' \
  mds 'allow rw path=/' \
  osd 'allow rw pool=cephfs.docker.meta, allow rw pool=cephfs.docker.data'
  -o /etc/pve/priv/ceph/ceph.client.docker-cephfs.keyring

2. Extract the raw secret

grep 'key =' /etc/pve/priv/ceph/ceph.client.docker-cephfs.keyring | awk '{print $3}' > /etc/pve/priv/ceph/docker-cephFS.secret
chmod 600 /etc/pve/priv/ceph/docker-cephFS.secret

3. Generate a minimal Ceph config

ceph config generate-minimal-conf -o /etc/pve/priv/ceph/minimal-ceph.conf
chmod 644 /etc/pve/priv/ceph/minimal-ceph.conf

🛠️ VM Setup Instructions (done within VM)

1. Install necessary packages

apt update
apt install ceph-common

2. Retrieve secret and config from Proxmox

sftp root@[fc00::81]
lcd ~
get /etc/pve/priv/ceph/docker-cephFS.secret
get /etc/pve/priv/ceph/minimal-ceph.conf
get /etc/pve/priv/ceph/ceph.client.docker-cephfs.keyring
exit

3. Move files into place

mkdir -p /etc/ceph
mv ~/docker-cephFS.secret /etc/ceph/
mv ~/minimal-ceph.conf /etc/ceph/ceph.conf
mv ~/ceph.client.docker-cephfs.keyring /etc/ceph/ceph.client.docker-cephfs.keyring
chmod 600 /etc/ceph/ceph.client.docker-cephfs.keyring
chmod 600 /etc/ceph/docker-cephFS.secretget 
chmod 644 /etc/ceph/ceph.conf

4. Create mount point

mkdir -p /mnt/docker-cephFS

5. Test manual mount

mount -t ceph :/ /mnt/docker-cephFS \
    -o name=docker-cephfs,secretfile=/etc/ceph/docker-cephFS.secret,conf=/etc/ceph/ceph.conf,fs=docker

6. Configure permanent mount in `/etc/fstab`

Add this line to /etc/fstab:

:/ /mnt/docker-cephFS ceph name=docker-cephfs,secretfile=/etc/ceph/docker-cephFS.secret,conf=/etc/ceph/ceph.conf,fs=docker,_netdev 0 2

🔥 Optional: Automated Bootstrap Script for New VMs

Create a file /root/cephfs-bootstrap.sh with the following contents:

#!/bin/bash

apt update
apt install -y ceph-common

mkdir -p /etc/ceph
mkdir -p /mnt/docker-cephFS

sftp root@[fc00::81] <<EOF
lcd /etc/ceph
get /etc/pve/priv/ceph/docker-cephFS.secret
get /etc/pve/priv/ceph/minimal-ceph.conf
bye
EOF

chmod 600 /etc/ceph/docker-cephFS.secret
chmod 644 /etc/ceph/minimal-ceph.conf
mv /etc/ceph/minimal-ceph.conf /etc/ceph/ceph.conf

mount -t ceph :/ /mnt/docker-cephFS \
    -o name=docker-cephfs,secretfile=/etc/ceph/docker-cephFS.secret,conf=/etc/ceph/ceph.conf,fs=docker

Make it executable:

chmod +x /root/cephfs-bootstrap.sh

Run it:

/root/cephfs-bootstrap.sh

✅ This script will install packages, pull configs, set permissions, and mount automatically!

🔒 Files Overview

File	Purpose
`/etc/pve/priv/ceph/ceph.client.docker-cephfs.keyring`	Full Ceph client keyring (admin level)
`/etc/pve/priv/ceph/docker-cephFS.secret`	Raw base64 secret for kernel mounting
`/etc/pve/priv/ceph/minimal-ceph.conf`	Clean minimal Ceph config

🚀 TL;DR

Pull secret + minimal conf from /etc/pve/priv/ceph/, mount :/ with fs=docker into /mnt/docker-cephFS. Use fstab for permanent mount.

This procedure is safe, clean, Proxmox-cluster aware, and scales easily across VMs.

scyto/docker-vm-ceph-setup.md