Docker Swarm Plugin - freenas-iscsi - unable to find iscsi session

[derekpovah]

It seems like iscsi is being properly set up by the plugin, but the plugin can't do anything with it once it's set up. iscsiadm in the container is wrapped to force a chroot, so my assumption is that the container should be able to see all of the iscsi sessions the host can see. At one point, I had all of the capabilities that @olljanat mentioned in #368 in the plugin config, but that didn't change anything, so I'm back to just CAP_SYS_ADMIN.

Here's what I'm doing to get to where I'm at:

• Build / install / enable the plugin based on https://github.com/olljanat/csi-plugins-for-docker-swarm
• Make sure to start in a clean state:
  • iscsiadm -m discoverydb -P1 on the host shows No targets found.
  • iscsiadm -m session on the host returns No active sessions.
  • lsblk doesn't show any additional disks
• docker volume create --driver csi-freenas-iscsi [...] nginx-iscsi does its magic and I see a zvol in TrueNAS.
• docker service create --mount type=cluster,src=nginx-iscsi [...] hangs forever.
• After creating the service:
  • iscsiadm -m discoverydb -P1 both inside and outside the plugin container shows the correct target
  • iscsiadm -m session on the host shows a session but within the container it shows No active sessions
  • lsblk now shows /dev/sdb both inside and outside the plugin container

I don't think this has anything to do with freenas-iscsi.yaml since zvols and iscsi targets are dynamically created and removed by the plugin without an issue. (Except for one hangup where the iscsi session isn't cleaned up by the plugin container since it can't see it. Once the sessions are manually logged out on the host, volumes are automatically removed by the plugin.)

Here's the plugin config:

{
    "description": "democratic-csi storage driver",
    "interface": {
      "types": ["docker.csinode/1.0", "docker.csicontroller/1.0"],
      "socket": "csi-freenas-iscsi.sock"
    },
    "network": {
      "type": "host"
    },
    "mounts": [
      {
        "name": "host",
        "description": "Allows iscsiadm to run in a chroot of the host filesystem",
        "source": "/",
        "destination": "/host",
        "type": "bind",
        "options": ["bind"]
      }
    ],
    "env": [
      {
        "name": "CSI_ENDPOINT",
        "value": "/run/docker/plugins/csi-freenas-iscsi.sock"
      }
    ],
    "entrypoint": [
      "/home/csi/app/entrypoint.sh"
    ],
    "workdir": "/home/csi/app",
    "linux": {
      "capabilities": [
        "CAP_SYS_ADMIN"
      ],
      "AllowAllDevices": true,
      "devices": null
    },
    "PropagatedMount": "/data/published"
  }

Side note: I built a plugin for freenas-nfs, filled out the sample configuration yaml and was able to create and mount an nfs share without any issues. Super cool!

[travisghansen]

Can you enable ipc=host on the container and see if that clears it up?

[derekpovah]

Even with "ipchost": true" in plugin's config.json, I still see rpc error: code = Unknown desc = unable to find iscsi session for iqn: [...] in the docker logs. The iscsi target and session is still being created successfully on the host, which is the behavior that's throwing me off here.

[travisghansen]

OK, set --pid=host as well. After setting that try enabling the nsenter strategy and see if that works.

ISCSIADM_HOST_STRATEGY=nsenter env var on the container

https://github.com/democratic-csi/democratic-csi/blob/master/docker/iscsiadm