dragonfly-operator: spec.additionalVolumeMounts usage patterns (emptyDir crash-protection, read-only warm-up cache, seed+emptyDir combo)

dragonfly-operator spec.additionalVolumeMounts usage patterns.md

dragonfly-operator: spec.additionalVolumeMounts usage patterns

spec.additionalVolumeMounts mounts extra volumes into the Dragonfly main container. It is the missing counterpart to the existing spec.additionalVolumes (which only adds volumes to the pod, without mounting them). Pairing the two lets you compose several snapshot/cache patterns without any snapshot-specific API — the operator does not need to know what the volume is for.

Status: additionalVolumeMounts is proposed in a fork of dragonfly-operator (based on v1.6.1). additionalVolumes, initContainers and spec.snapshot.dir are existing v1.6.1 fields. All examples below were validated end-to-end on a real cluster (Dragonfly v1.27.0, operator v1.6.1 + the field).

Background: why this matters

• In-place container restarts lose data. A kubelet minor upgrade that changes the container hash (e.g. 1.30→1.31) restarts every container on the node in place (same pod, restartCount++). Without a snapshot, the Dragonfly master comes back empty and a replica full-syncs from the empty master → both empty.
• Dragonfly saves a snapshot on SIGTERM to --dir and loads the latest snapshot from --dir on start. So a snapshot dir that survives the container restart is enough to preserve data.
• emptyDir is ideal for that: it survives in-place container restarts (pod is alive → volume is alive), does not pin the pod to a node (unlike a local-path PVC), and is free.
• Operator v1.6.1 adds --break_replication_on_master_restart=true by default, which alone protects the rolling case (nodes upgraded one at a time — a restarted empty master no longer poisons the surviving replica). A snapshot volume additionally protects the simultaneous both-pods restart / hard crash of both.

Pattern picker

You want…	Use	File
Survive in-place restarts / simultaneous crash without pinning the pod	writable emptyDir as snapshot dir	pattern-1-emptydir-crash-protection.yaml
Immutable warm-up cache: every start loads a golden snapshot, runtime writes never touch it	readOnly seed volume as snapshot dir	pattern-2-readonly-warmup-cache.yaml
Warm up from a golden seed and keep runtime state across in-place restarts	init-seed → writable emptyDir	pattern-3-combo-seed-plus-emptydir.yaml

Verified behavior

Scenario	Pattern 1 emptyDir	Pattern 2 readOnly seed	Pattern 3 combo
Fresh pod startup	starts empty (or replicates)	loads golden seed	loads golden seed (init)
Runtime writes	persist to emptyDir on save	in-memory only, never hit disk	persist to emptyDir on save
In-place container restart	✅ reloads saved state	✅ reloads golden seed	✅ reloads grown state
Both pods restart at once	✅ each reloads own snapshot	n/a (warm-up)	✅ each reloads own snapshot
Pod reschedule (node death)	fresh emptyDir → replica resync	re-loads golden seed	re-seeds golden baseline

Verified on a live cluster

Ran on a real cluster (Dragonfly v1.27.0, 1000-key dataset), triggering an in-place container restart with kill -TERM 1 (same effect as a kubelet container-hash restart):

Scenario	Check	Result
S1 — emptyDir	restart of both containers at once	✅ 1000/1000
S2 — read-only warm-up	loads seed / runtime write gone after restart	✅ 1000 → write → restart → 1000, k:HACK absent
S3 — combo	warm-up → grow → container restart → pod reschedule	✅ 1000 → 1002 → 1002 (growth kept) → 1000 (re-seed)

Notes / gotchas

• additionalVolumeMounts is a plain []corev1.VolumeMount, so subPath, readOnly, mountPropagation etc. are available — set only what you need.
• readOnly + snapshot.cron don't mix: cron/SIGTERM saves log Failed to perform snapshot Read-only file system (a warning, not a crash). Omit cron on read-only snapshot dirs.
• emptyDir survives container restart, not pod deletion/reschedule — it is runtime protection, not durable backup. For durable snapshots use a PVC or S3.
• The combo pattern relies on init containers running once per pod creation, not per container restart, and on Dragonfly always loading the latest dump in --dir (runtime saves are newer than the seed).

pattern-1-emptydir-crash-protection.yaml

# Pattern 1 — writable emptyDir as the snapshot dir (crash / kubelet-upgrade protection)
#
# Dragonfly SAVEs to /dragonfly/snapshots on SIGTERM and reloads it on start.
# emptyDir survives in-place container restarts (kubelet container-hash change)
# without pinning the pod to a node the way a local-path PVC does.
#
# Verified: kill -TERM 1 in one or both pods -> data preserved (1000/1000).
apiVersion: dragonflydb.io/v1alpha1
kind: Dragonfly
metadata:
  name: df-emptydir
spec:
  replicas: 2
  # spec.snapshot.dir just sets --dir; the volume+mount come from the generic fields.
  snapshot:
    dir: /dragonfly/snapshots
    cron: "*/5 * * * *"          # optional periodic save; SIGTERM already saves on shutdown

  additionalVolumes:
    - name: snapshots
      emptyDir: {}               # survives container restart, not pod reschedule

  additionalVolumeMounts:
    - name: snapshots
      mountPath: /dragonfly/snapshots   # writable: Dragonfly saves & reloads here

pattern-2-readonly-warmup-cache.yaml

# Pattern 2 — read-only golden snapshot as an immutable warm-up cache
#
# Every start loads a pre-baked "golden" snapshot. Runtime writes stay in memory
# and can NEVER corrupt the golden copy: the filesystem is mounted read-only, so
# the SIGTERM/cron save fails harmlessly (a warning, not a crash) and the instance
# always warms back up to the exact same baseline.
#
# Use for caches you rebuild from source of truth: fast startup from a known-good
# dataset, zero risk of poisoning it.
#
# Verified: loads 1000 keys on start; a runtime `set` is gone after restart; the
# on-disk seed is never modified (EROFS).
#
# Provision the golden snapshot first, e.g.:
#   - a Job that runs Dragonfly with --dir + a snapshot and writes dump-*.dfs, or
#   - copy dump-*.dfs from an existing instance into the volume.
# Here it lives on a hostPath for illustration — a read-only PVC or CSI volume works too.
apiVersion: dragonflydb.io/v1alpha1
kind: Dragonfly
metadata:
  name: df-warmup
spec:
  replicas: 1
  # nodeSelector only needed because the example seed is a node-local hostPath.
  nodeSelector:
    kubernetes.io/hostname: worker-01

  snapshot:
    dir: /dragonfly/snapshots
    # NO cron here: the dir is read-only, cron saves would just log warnings.

  additionalVolumes:
    - name: seed
      hostPath:
        path: /opt/df-seed        # directory holding the golden dump-*.dfs
        type: Directory

  additionalVolumeMounts:
    - name: seed
      mountPath: /dragonfly/snapshots
      readOnly: true              # golden copy cannot be modified by the instance

pattern-3-combo-seed-plus-emptydir.yaml

# Pattern 3 — golden seed warm-up + writable emptyDir (combined)
#
# Best of both: warm up fast from a golden baseline on a fresh pod, AND keep the
# live runtime state across in-place container restarts.
#
# How it works:
#   - initContainer copies the golden seed into a writable emptyDir. Init containers
#     run ONCE per pod creation, NOT per container restart.
#   - Dragonfly's --dir is that writable emptyDir: it loads the LATEST dump there and
#     saves on SIGTERM/cron.
#   => fresh pod: init seeds -> load baseline.
#   => in-place container restart (kubelet upgrade): init does NOT re-run, emptyDir
#      persists the saved (grown) state -> reload grown state.
#   => pod reschedule (node death): fresh emptyDir -> init re-seeds baseline
#      (then replication catches up for an HA pair).
#
# Verified: fresh=1000; +2 keys -> restart container -> 1002 (grown state kept);
# delete pod -> back to 1000 (re-seeded baseline).
apiVersion: dragonflydb.io/v1alpha1
kind: Dragonfly
metadata:
  name: df-combo
spec:
  replicas: 1
  image: docker.dragonflydb.io/dragonflydb/dragonfly:v1.27.0
  nodeSelector:
    kubernetes.io/hostname: worker-01   # only because the example seed is a hostPath

  snapshot:
    dir: /dragonfly/snapshots
    cron: "*/5 * * * *"          # fine here: the working dir is writable

  additionalVolumes:
    - name: work                 # writable runtime snapshot dir
      emptyDir: {}
    - name: seed                 # golden baseline (read-only source)
      hostPath:
        path: /opt/df-seed
        type: Directory

  initContainers:
    - name: seed-warmup
      image: docker.dragonflydb.io/dragonflydb/dragonfly:v1.27.0
      # -n (no-clobber): only fills an empty emptyDir; a grown emptyDir on a
      # surviving pod is never touched (init doesn't run on container restart anyway).
      command: ["sh", "-c", "cp -an /seed/. /work/ 2>/dev/null || true"]
      volumeMounts:
        - name: seed
          mountPath: /seed
          readOnly: true
        - name: work
          mountPath: /work

  additionalVolumeMounts:
    - name: work
      mountPath: /dragonfly/snapshots   # main container: writable emptyDir