Skip to content

Instantly share code, notes, and snippets.

@ProjectInitiative

ProjectInitiative/setup.md

Last active August 15, 2025 00:04
Show Gist options
  • Save ProjectInitiative/3a4c9ea03a0ebcc585fe3f008f93cd24 to your computer and use it in GitHub Desktop.
Save ProjectInitiative/3a4c9ea03a0ebcc585fe3f008f93cd24 to your computer and use it in GitHub Desktop.
Download ZIP
Bcachefs 6.17 RC Current Setup
Raw
setup.md

Bcachefs Usage Report – 3-Node Kubernetes Cluster on NixOS

Setup Overview

I run a 3-node Kubernetes cluster on NixOS, with bcachefs as the primary storage pool.
Bcachefs handles auto re-replication, scrubbing, and integrated stats/notifications get sent to telegram (and eventually grafana).

Underlying Storage Layout

Main Cluster Storage

  • Cache layer: 2 TB NVMe
  • Read/promote cache: 1 × SATA SSD
  • Primary capacity:
    • 2 × 16 TB HDDs (CMR)
    • 1 × 6 TB HDD (CMR, legacy)
  • Previous setup: 3 × 6 TB SMR drives
    • Successfully live-migrated the pool from SMRs to the current all-CMR setup with zero downtime — a very cool moment.

Repurposed Old Drives

  • Old SMRs now live in a low-spec NAS used for homelab backups.
  • NAS storage: 1 × 1 TB NVMe + 6 × 6 TB SMR drives.
  • Even with SMRs, performance is surprisingly decent thanks to bcachefs’s COW nature.

RAW Total Storage:

  • k8s cluster: 123TB (41TB per node)
  • Backup NAS: 37TB

Usable Total Storage:

  • Individual nodes: 20.5TB (Replicas 2)
  • Backup NAS: 18.5TB (Replicas 2)

Workloads on the Kubernetes Cluster

1. App-Local CSI Storage

  • Filesystem-level option for local CSI volumes ensures certain workloads (e.g., distributed PostgreSQL) always store data on the cache layer.

2. Generic CSI Storage

  • Lets bcachefs handle tiering and transparent data movement.
  • Example: Self-hosted S3 cluster (Garage)
    • Hot objects (e.g., Immich thumbnails, media files) are automatically promoted to faster tiers.
    • S3 reads become much faster without any application-level awareness.

3. Nix Binary Cache

  • Nix binary cache serves the entire homelab.
  • Makes rebuilding Kent’s kernel tree and distributing it to all hosts trivial.
  • S3 server’s deduplication + bcachefs background compression = highly space-efficient.

Other Uses

  • Developer Laptop Home Directory:
    • Snapshots have completely replaced backintime backups to another partition.
      • This proved useful as FreeCAD corrupted a few open project files. Did lose any work as I was able to pull directly from the snapshot list
    • Saves space and simplifies backup management.
  • Despite some crashes and mount failures in the distant past, no data loss has occurred.

Impressions

Bcachefs has proven to be flexible, powerful, and genuinely fun to work with.

Highlights:

  • Live migration from SMR → CMR drives with no downtime.
  • Transparent tiering for mixed workloads.
  • Deduplication + compression synergy with S3 storage.
  • Snapshots making legacy backup tools obsolete.
  • Despite my issues with newer kernels and NVME power management knocking NVMEs offline constantly, I would have never known if I didn't see the performance degradation and notifications as the FS handled the failures gracefully.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment