Talos Linux democratic-csi nfs and iscsi with Truenas Scale

README.md

How To Deploy

The .yaml file is actually a values file for the Helm chart democratic-csi/democratic-csi

• Make sure TrueNAS has a dataset nvme2tb/k8s (or update the yaml file)
• If you're using iSCSI, do the following in TrueNAS:
  • Create a dataset nvme2tb/k8s/iscsi
  • Make sure Block (iSCSI) Shares Targets is running, and click Configure
  • Save the defaults for Target Global Configuration
  • Add a portal on 0.0.0.0:3260
  • Add an Initiator Group, Allow all initiators, and name it something like k8s-talos
  • Create a Target named donotdelete and alias donotdelete, then add iSCSI group selecting the Portal and Initiator Group you just created. This prevents TrueNAS from deleting the Initiator Group if you're testing and you delete the one and only PV.
  • Make note of the portal ID and the Initiator Group ID and update these values in the file freenas-api-iscsi.yaml if needed
  • During testing, the manually created Initiator Group was getting deleted whenever deleting the last PV. This appears to be a bug in TrueNAS somewhere according to democratic-csi/democratic-csi#412. Essentially TrueNAS deletes the Initiator Group automatically if an associated Target is deleted and no others exist. If you followed the instructions and created a manual Target this won't be an issue :)
• Make sure TrueNAS has an API key that matches what's in the yaml file
  • If not, add one in TrueNAS in the top right corner go to Admin > API Keys
• Create the democratic-csi namespace: kubectl create ns democratic-csi
• Make that namespace privileged: kubectl label --overwrite namespace democratic-csi pod-security.kubernetes.io/enforce=privileged
• Deploy the helm chart for each type of storage class you want:
  • helm upgrade --install --namespace democratic-csi --values freenas-api-nfs.yaml truenas-nfs democratic-csi/democratic-csi
  • helm upgrade --install --namespace democratic-csi --values freenas-api-iscsi.yaml truenas-iscsi democratic-csi/democratic-csi
  • (WIP, this works up until attempting to attach a pod to the PVC and I can't get support on it)helm upgrade --install --namespace democratic-csi --values zfs-generic-nvmeof.yaml truenas-nvmeof democratic-csi/democratic-csi

Testing

• Check that the pods are running on each worker node: kubectl get po -n democratic-csi
  • If not, check why: kubectl describe pod [pod-name] -n democratic-csi
• Deploy the test pvc and wait a minute to verify it was provisioned
• Check with kubectl get pv
• If it's not working, check kubectl describe pvc testpvc

freenas-api-iscsi.yaml

driver:
  config:
    driver: freenas-api-iscsi
    httpConnection:
      protocol: https
      apiKey: shhhhhhh
      host: 10.0.50.99
      port: 443
      allowInsecure: true
    zfs:
      datasetParentName: nvme2tb/k8s/iscsi/volumes
      detachedSnapshotsDatasetParentName: nvme2tb/k8s/iscsi/snapshots
      zvolCompression:
      zvolDedup:
      zvolEnableReservation: false
      zvolBlockSize:
    iscsi:
      targetPortal: "10.0.50.99:3260"
      targetPortals: []
      interface:
      namePrefix: csi-
      nameSuffix: "-talos"
      targetGroups:
        - targetGroupPortalGroup: 1
          targetGroupInitiatorGroup: 5
          targetGroupAuthType: None
          targetGroupAuthGroup:
      extentInsecureTpc: true
      extentXenCompat: false
      extentDisablePhysicalBlocksize: true
      extentBlocksize: 512
      extentRpm: "SSD"
      extentAvailThreshold: 0
 
csiDriver:
  # should be globally unique for a given cluster
  name: "org.democratic-csi.freenas-api-iscsi"
 
storageClasses:
  - name: truenas-iscsi
    defaultClass: true
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    allowVolumeExpansion: true
    parameters:
      fsType: ext4
      detachedVolumesFromSnapshots: "false"
    mountOptions: []
    secrets:
      provisioner-secret:
      controller-publish-secret:
      node-stage-secret:
      node-publish-secret:
      controller-expand-secret:
 
volumeSnapshotClasses:
  - name: truenas-iscsi
    parameters:
      detachedSnapshots: "true"

node:
  hostPID: true
  driver:
    extraEnv:
      - name: ISCSIADM_HOST_STRATEGY
        value: nsenter
      - name: ISCSIADM_HOST_PATH
        value: /usr/local/sbin/iscsiadm
    iscsiDirHostPath: /usr/local/etc/iscsi
    iscsiDirHostPathType: ""

freenas-api-nfs.yaml

driver:
  config:
    driver: freenas-api-nfs
    httpConnection:
      protocol: https
      apiKey: shhhhhhh
      host: 10.0.50.99
      port: 443
      allowInsecure: true
    zfs:
      datasetParentName: nvme2tb/k8s
      detachedSnapshotsDatasetParentName: nvme2tb/snaps
      datasetEnableQuotas: true
      datasetEnableReservation: false
      datasetPermissionsMode: "0777"
      datasetPermissionsUser: 0
      datasetPermissionsGroup: 0
    nfs:
      shareCommentTemplate: "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}-{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
      shareHost: 10.0.50.99
      shareAlldirs: false
      shareAllowedHosts: []
      shareAllowedNetworks: []
      shareMaprootUser: root
      shareMaprootGroup: root
      shareMapallUser: ""
      shareMapallGroup: ""
 
csiDriver:
  # should be globally unique for a given cluster
  name: "org.democratic-csi.freenas-api-nfs"
 
storageClasses:
  - name: truenas-nfs
    defaultClass: false
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    allowVolumeExpansion: true
    parameters:
      fsType: nfs
    mountOptions:
      - noatime
      - nfsvers=4
 
volumeSnapshotClasses:
  - name: truenas