NFS Cluster in Kubernets

config.md

Configuration snippets for Highly Available NFS cluster in Kubernetes, a cloud vendor independent storage solution

HAProxy config

frontend nfs
    bind 192.168.1.100:32049
    mode tcp
    option tcplog
    default_backend nfs

backend nfs
    mode tcp
    option tcplog
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 3 slowstart 60s maxconn 250 maxqueue 256 weight 100
        server k8s-worker1 192.168.2.10:32049 check send-proxy
        server k8s-worker2 192.168.2.11:32049 check send-proxy
        server k8s-worker3 192.168.2.12:32049 check send-proxy

---

Traefik Config

ports:
  nfs:
    port: 2049
    expose: true
    exposedPort: 32049
    protocol: TCP
    nodePort: 32049

service:
  enabled: true
  type: NodePort
  spec:
    externalTrafficPolicy: Local

additionalArguments:
  - "--entryPoints.nfs.proxyProtocol.trustedIPs=192.168.1.2,192.168.1.3,192.168.1.100"

---

Traefik Ingress config

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: nfs-ingress
  namespace: rook-ceph
spec:
  entryPoints:
    - nfs
  routes:
    - match: HostSNI(`*`)
      middlewares:
      - name: nfs-ipwhitelist
        namespace: rook-ceph
      services:
      - name: rook-ceph-nfs-my-nfs-a
        port: 2049
      - name: rook-ceph-nfs-my-nfs-b
        port: 2049
      - name: rook-ceph-nfs-my-nfs-c
        port: 2049

---
apiVersion: traefik.containo.us/v1alpha1
kind: MiddlewareTCP
metadata:
  name: nfs-ipwhitelist
  namespace: rook-ceph
spec:
  ipWhiteList:
    sourceRange:
      - 192.168.3.100/32 # test
      - 192.168.4.10/32  # other cluster nodes
      - 192.168.4.11/32
      - 192.168.4.12/32
      - 192.168.5.10/32

Rook-Ceph

As of today 11/1/2021, I confirm this combination works fine for NFS.
Rook v1.7.6
Ceph v15.2.15

But this combination does not work, e.g. at the last step of "exporting", nothing could be exported.
Rook v1.7.6
Ceph v16.2.6

Config files to apply

Base URL
https://github.com/rook/rook/tree/v1.7.6/cluster/examples/kubernetes/ceph

kubectl apply -f crds.yaml
kubectl apply -f common.yaml
kubectl apply -f operator.yaml

// this file is manually updated to use Ceph v15.2.15
kubectl apply -f cluster.yaml

kubectl apply -f toolbox.yaml

// Ceph filesystem
kubectl apply -f filesystem.yaml

// nfs
kubectl apply -f nfs.yaml

// allow nfs export creation from dashboard
ceph dashboard set-ganesha-clusters-rados-pool-namespace myfs-data0/nfs-ns

kubectl apply -f storageclass.yaml

---