FreeNASSshDriver.probe failed

[ffzzhong]

hi, i'm installing democratic-sci helm chart version 0.14.5 with my customized values file,

the nodes can run well, but the csi-controler goes to CrashLoopBackOff status,

of these 6 containers of csi-controller pod, csi-driver and csi-proxy are running, while the other 4 external-* containers are crashing

if I check the external-* containers, I found it's all because of

I0402 17:00:29.213063       1 connection.go:164] Connecting to unix:///csi-data/csi.sock                                                                                                                       │
│ I0402 17:00:29.214053       1 common.go:111] Probing CSI driver for readiness                                                                                                                                  │
│ I0402 17:00:29.214113       1 connection.go:193] GRPC call: /csi.v1.Identity/Probe                                                                                                                             │
│ I0402 17:00:29.214150       1 connection.go:194] GRPC request: {}                                                                                                                                              │
│ I0402 17:00:29.239002       1 connection.go:200] GRPC response: {}                                                                                                                                             │
│ I0402 17:00:29.239018       1 connection.go:201] GRPC error: rpc error: code = Internal desc = TypeError: err.getMessage is not a function TypeError: err.getMessage is not a function                         │
│     at FreeNASSshDriver.Probe (/home/csi/app/src/driver/freenas/ssh.js:49:46)                                                                                                                                  │
│     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)                                                                                                                              │
│     at async requestHandlerProxy (/home/csi/app/bin/democratic-csi:222:18)                                                                                                                                     │
│ E0402 17:00:29.239045       1 main.go:141] CSI driver probe failed: rpc error: code = Internal desc = TypeError: err.getMessage is not a function TypeError: err.getMessage is not a function                  │
│     at FreeNASSshDriver.Probe (/home/csi/app/src/driver/freenas/ssh.js:49:46)                                                                                                                                  │
│     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)                                                                                                                              │
│     at async requestHandlerProxy (/home/csi/app/bin/democratic-csi:222:18)

I believe it's because of the SSH check, by referring some previous github issues:

• understand the actual log comes from csi-driver container, it's saying {"host":"zfs-nfs-democratic-csi-controller-698d858585-q4nw2","level":"error","message":"handler error - driver: FreeNASSshDriver method: Probe error: TypeError: err.getMessage is not a function TypeError, if I enable debug logging for it, it says:

{"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"info","message":"new request - driver: FreeNASSshDriver method: Probe call: {\"path\":\"/csi.v1.Identity/Probe\",\"metadata\":{},\"reque │
│ {"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"debug","message":"FREENAS HTTP REQUEST DETAILS: {\"headers\":{\"Accept\":\"application/json\",\"User-Agent\":\"democratic-csi-driver\",\ │
│ {"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"debug","message":"FREENAS HTTP REQUEST DURATION: 0.01s","service":"democratic-csi","timestamp":"2024-04-02T17:11:17.613Z"}               │
│ {"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"debug","message":"FREENAS HTTP ERROR: null","service":"democratic-csi","timestamp":"2024-04-02T17:11:17.614Z"}                           │
│ {"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"debug","message":"FREENAS HTTP RESPONSE STATUS CODE: 401","service":"democratic-csi","timestamp":"2024-04-02T17:11:17.614Z"}             │
│ {"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"debug","message":"FREENAS HTTP RESPONSE HEADERS: {\"server\":\"nginx\",\"date\":\"Tue, 02 Apr 2024 17:11:17 GMT\",\"content-type\":\"tex │
│ {"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"debug","message":"FREENAS HTTP RESPONSE BODY: \"401: Unauthorized\"","service":"democratic-csi","timestamp":"2024-04-02T17:11:17.614Z"}  │
│ {"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"debug","message":"FREENAS HTTP REQUEST DETAILS: {\"headers\":{\"Accept\":\"application/json\",\"User-Agent\":\"democratic-csi-driver\",\ │
│ {"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"debug","message":"FREENAS HTTP REQUEST DURATION: 0.01s","service":"democratic-csi","timestamp":"2024-04-02T17:11:17.621Z"}               │
│ {"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"debug","message":"FREENAS HTTP ERROR: null","service":"democratic-csi","timestamp":"2024-04-02T17:11:17.621Z"}                           │
│ {"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"debug","message":"FREENAS HTTP RESPONSE STATUS CODE: 200","service":"democratic-csi","timestamp":"2024-04-02T17:11:17.621Z"}             │
│ {"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"debug","message":"FREENAS HTTP RESPONSE HEADERS: {\"server\":\"nginx\",\"date\":\"Tue, 02 Apr 2024 17:11:17 GMT\",\"content-type\":\"tex │
│ {"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"debug","message":"FREENAS HTTP RESPONSE BODY: \"<!doctype html>\\n<html lang=\\\"en\\\">\\n<head>\\n  <meta charset=\\\"utf-8\\\">\\n  < │
│ {"host":"zfs-nfs-democratic-csi-controller-698d858585-656gw","level":"error","message":"handler error - driver: FreeNASSshDriver method: Probe error: TypeError: err.getMessage is not a function TypeError:

• I can confirm that I can ping the NAS server IP from another debug pod , I can also ping the NAS server IP from the node that runs the debug pod
• in my values file, I tried with root user, non-root user, also tried providing the password or the private key, none of them get through, however, I can successfully connect to NAS easily from another machine

pls help to point out what could the issue and how to solve it, thx!

[travisghansen]

I'll fix that and maybe then we can see what the error is.

[Zi-SH]

I can confirm that I'm seeing the same issue on chart version 0.14.5 with csi-driver and csi-proxy running, and the same error on the external-* containers:

I0410 18:31:01.936534       1 feature_gate.go:249] feature gates: &{map[]}
I0410 18:31:01.937326       1 connection.go:164] Connecting to unix:///csi-data/csi.sock
I0410 18:31:01.938279       1 common.go:138] Probing CSI driver for readiness
I0410 18:31:01.938302       1 connection.go:193] GRPC call: /csi.v1.Identity/Probe
I0410 18:31:01.938308       1 connection.go:194] GRPC request: {}
I0410 18:31:01.961035       1 connection.go:200] GRPC response: {}
I0410 18:31:01.961063       1 connection.go:201] GRPC error: rpc error: code = Internal desc = TypeError: err.getMessage is not a function TypeError: err.getMessage is not a function
    at FreeNASSshDriver.Probe (/home/csi/app/src/driver/freenas/ssh.js:49:46)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async requestHandlerProxy (/home/csi/app/bin/democratic-csi:222:18)
F0410 18:31:01.961092       1 main.go:134] failed probing CSI driver: CSI driver probe failed: rpc error: code = Internal desc = TypeError: err.getMessage is not a function TypeError: err.getMessage is not a function
    at FreeNASSshDriver.Probe (/home/csi/app/src/driver/freenas/ssh.js:49:46)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async requestHandlerProxy (/home/csi/app/bin/democratic-csi:222:18)```

[Zi-SH]

I can confirm dropping democratic-csi back to democratic-csi:v1.8.4 in the values.yaml results in the system working as expected.

[travisghansen]

Sorry for the inconvenience. I'll get this cleaned up and new release out with a fix.

[travisghansen]

Should be fixed here: 38bee21

Give v1.9.1 a try.