Bcachefs Usage Report – 3-Node Kubernetes Cluster on NixOS

Setup Overview

I run a 3-node Kubernetes cluster on NixOS, with bcachefs as the primary storage pool.
Bcachefs handles auto re-replication, scrubbing, and integrated stats/notifications get sent to telegram (and eventually grafana).

Underlying Storage Layout

Main Cluster Storage

Cache layer: 2 TB NVMe
Read/promote cache: 1 × SATA SSD
Primary capacity:
- 2 × 16 TB HDDs (CMR)
- 1 × 6 TB HDD (CMR, legacy)
Previous setup: 3 × 6 TB SMR drives
- Successfully live-migrated the pool from SMRs to the current all-CMR setup with zero downtime — a very cool moment.

Repurposed Old Drives

Old SMRs now live in a low-spec NAS used for homelab backups.
NAS storage: 1 × 1 TB NVMe + 6 × 6 TB SMR drives.
Even with SMRs, performance is surprisingly decent thanks to bcachefs’s COW nature.

RAW Total Storage:

k8s cluster: 123TB (41TB per node)
Backup NAS: 37TB

Usable Total Storage:

Individual nodes: 20.5TB (Replicas 2)
Backup NAS: 18.5TB (Replicas 2)

Workloads on the Kubernetes Cluster

1. App-Local CSI Storage

Filesystem-level option for local CSI volumes ensures certain workloads (e.g., distributed PostgreSQL) always store data on the cache layer.

2. Generic CSI Storage

Lets bcachefs handle tiering and transparent data movement.
Example: Self-hosted S3 cluster (Garage)
- Hot objects (e.g., Immich thumbnails, media files) are automatically promoted to faster tiers.
- S3 reads become much faster without any application-level awareness.

3. Nix Binary Cache

Nix binary cache serves the entire homelab.
Makes rebuilding Kent’s kernel tree and distributing it to all hosts trivial.
S3 server’s deduplication + bcachefs background compression = highly space-efficient.

Other Uses

Developer Laptop Home Directory:
- Snapshots have completely replaced backintime backups to another partition.
Despite some crashes and mount failures in the distant past, no data loss has occurred.
Snapshots for Fast Recovery
- Snapshots have completely replaced my reliance on a separate partition for backups. This was recently proven invaluable when FreeCAD corrupted a few project files. Thanks to the integrated snapshot feature, I was able to recover my work directly from the snapshot list, losing no data.
- Snapshots provide an excellent, space-efficient way to recover from accidental deletions or file corruption, significantly improving my Recovery Point Objective (RPO).

Important Note on Backups

While snapshots are powerful for rapid recovery, it's crucial to understand they are not a substitute for a full backup strategy. They reside on the same storage as the live data. For true data resiliency, it's essential to follow the 3-2-1 backup rule:

3: copies of your data.

2: different storage types.

1: off-site copy.

For my homelab, the old SMR drives now serve as a low-spec NAS, providing a separate location for a second copy of critical data, which aligns with a piece this strategy (NAS has a copy on bcachefs, and there is a separate device running a different FS to mirror that as well.

Impressions

Bcachefs has proven to be flexible, powerful, and genuinely fun to work with.

Highlights:

Live migration from SMR → CMR drives with no downtime.
Transparent tiering for mixed workloads.
Deduplication + compression synergy with S3 storage.
Snapshots making legacy backup tools obsolete.
Despite my issues with newer kernels and NVME power management knocking NVMEs offline constantly, I would have never known if I didn't see the performance degradation and notifications as the FS handled the failures gracefully.

ProjectInitiative/setup.md