Unable to clone TrueNAS iSCSI volumes

[perfectra1n]

Hi there!

Love the project, I love it for my NFS shares. I'm also trying to use iSCSI with it, and have everything set up and running. The only issue is that when I try to take backups using Volsync, they're empty except for only containing the folder /lost+found.

However, when looking at the folder, they certainly aren't empty:

So this led me down the rabbit hole of permissons, etc., where I eventually cloned the PVC using the following:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vaultwarden-test-pvc
  namespace: apps
spec:
  storageClassName: "truenas-csi-iscsi"
  accessModes:
    - ReadWriteOnce
  dataSource:
    kind: PersistentVolumeClaim
    name: vaultwarden-main-pvc
  resources:
    requests:
      storage: 10Gi

And the "source" PVC:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vaultwarden-main-pvc
  namespace: apps
spec:
  storageClassName: "truenas-csi-iscsi"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

And the resulting volume only contained /lost+found.

Here's the ArgoCD Application that I've deployed for democratic-csi:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: democratic-iscsi-csi
  namespace: argocd
spec:
  project: default
  source:
    chart: democratic-csi
    repoURL: https://democratic-csi.github.io/charts
    targetRevision: "0.14.1"
    helm:
      releaseName: democratic-iscsi-csi-release
      values: |
        csiDriver:
          # should be globally unique for a given cluster
          name: "org.main-iscsi-democratic-csi.iscsi"

        storageClasses:
        - name: truenas-csi-iscsi
          defaultClass: false
          # Retain them instead of deleting them
          reclaimPolicy: Retain
          volumeBindingMode: Immediate
          allowVolumeExpansion: true
          parameters:
            fsType: ext4
        
        volumeSnapshotClasses:
          - name: main-truenas-iscsi-vsc
            deletionPolicy: Delete
            parameters:
              detachedSnapshots: "true"
        
        driver:
          existingConfigSecret: main-iscsi-democratic-csi-secrets
          config:
            driver: freenas-iscsi
  destination:
    server: "https://kubernetes.default.svc"
    namespace: storage

And here's the existingConfigSecret for it:

apiVersion: v1
kind: Secret
metadata:
 name: main-iscsi-democratic-csi-secrets
 namespace: storage
type: Opaque
stringData:
 driver-config-file.yaml: |
   driver: freenas-iscsi
   httpConnection:
     protocol: http
     host: nas.domain.network
     port: 80
     allowInsecure: true
     apiKey: apikey123
     apiVersion: 2
   sshConnection:
     host: nas.domain.network
     port: 22
     username: root
     # use either password or key
     password: ""
     privateKey: |
       -----BEGIN OPENSSH PRIVATE KEY-----
       .....
       -----END OPENSSH PRIVATE KEY-----

   zfs:
     datasetParentName: main_pool/kube_iscsi_volumes
     # This may be an issue in the future
     #datasetEnableQuotas: true
     detachedSnapshotsDatasetParentName: main_pool/kube_iscsi_volumes_backups
   iscsi:
     targetPortal: "192.168.2.155:3260"
     # for multipath
     targetPortals: ["192.168.2.155:3260"] # [ "server[:port]", "server[:port]", ... ]
     # leave empty to omit usage of -I with iscsiadm
     interface:

     # MUST ensure uniqueness
     # full iqn limit is 223 bytes, plan accordingly
     # default is "{{ name }}"
     nameTemplate: "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}-{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
     namePrefix: csi-
     nameSuffix: "-iscsivol"

     # add as many as needed
     targetGroups:
       # get the correct ID from the "portal" section in the UI
       - targetGroupPortalGroup: 1
         # get the correct ID from the "initiators" section in the UI
         targetGroupInitiatorGroup: 4
         # None, CHAP, or CHAP Mutual
         targetGroupAuthType: None
         # get the correct ID from the "Authorized Access" section of the UI
         # only required if using Chap
         targetGroupAuthGroup:

     extentCommentTemplate: "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}/{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
     extentInsecureTpc: true
     extentXenCompat: false
     extentDisablePhysicalBlocksize: true
     # 512, 1024, 2048, or 4096,
     extentBlocksize: 4096
     # "" (let FreeNAS decide, currently defaults to SSD), Unknown, SSD, 5400, 7200, 10000, 15000
     extentRpm: "SSD"
     # 0-100 (0 == ignore)
     extentAvailThreshold: 0

Is there anything obvious that I'm doing wrong that might lead to cloned PVCs being empty, or something absolutely boneheaded that could cause this? The truenas-csi-iscsi StorageClass is able to provision volumes just fine, move them around between nodes, etc. I don't see anything obvious stand out in the container logs, but I'm more than happy to provide some if there were some additional troubleshooting steps that I could perform to trigger some interesting logs.

Thanks again for the project, I love it and I appreciate your time.

[travisghansen]

I’m not ignoring you :) just currently traveling and this will take a bit to dig into to understand what’s happening.

[perfectra1n]

It's all good! I knew you never loved me </3 /s

Travel safely! Performing non-clone backups in Volsync using the iSCSI driver work just fine, so I at least have that workaround :)