[BUG] Attach failed for volume

[ambujjain]

I need some guidance to overcome an issue we are facing. We are attempting to deploy Longhorn in Ubuntu 22.04 with k3s, but we are encountering issues where volume attachment is timing out. We are unsure of the exact issue, but when we check the syslog, we see the following errors.

Events:
Type Reason Age From Message

---

Normal Scheduled 34m default-scheduler Successfully assigned default/psql-postgresql-0 to vm3
Warning FailedAttachVolume 2m1s (x24 over 34m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-f4ce40f3-27b4-41dc-961b-184a5es

[ambujjain]

Attching support bundle
Uploading supportbundle_b7f5b89b-7a71-41fb-aff3-0a148037c8c1_2024-05-16T03-01-46Z.zip…

[PhanLe1010]

Attching support bundle Uploading supportbundle_b7f5b89b-7a71-41fb-aff3-0a148037c8c1_2024-05-16T03-01-46Z.zip…

The support bundle link seems broken. Could you double check @ambujjain

[PhanLe1010]

Btw, you can also look into the longhorn-csi and longhorn-manager logs for more clue

[ambujjain]

I tried to upload bundle with longhorn portal.. and it was completed successfully.
how can i check longhorn-csi and longhorn-manager logs.. could you please guide us.

[vrevanth007]

Logs and support bundle of above mentioned cluster

root@vm1:~# kubectl logs -f -n longhorn-system   longhorn-csi-plugin-smgct -p
Defaulted container "node-driver-registrar" out of: node-driver-registrar, longhorn-liveness-probe, longhorn-csi-plugin
I0507 11:29:02.116312    2600 main.go:135] Version: v2.9.2
I0507 11:29:02.116570    2600 main.go:136] Running node-driver-registrar in mode=
I0507 11:29:02.116588    2600 main.go:157] Attempting to open a gRPC connection with: "/csi/csi.sock"
W0507 11:29:12.117540    2600 connection.go:232] Still connecting to unix:///csi/csi.sock
I0507 11:29:17.881790    2600 main.go:164] Calling CSI driver to discover driver name
I0507 11:29:17.895352    2600 main.go:173] CSI driver name: "driver.longhorn.io"
I0507 11:29:17.895420    2600 node_register.go:55] Starting Registration Server at: /registration/driver.longhorn.io-reg.sock
I0507 11:29:17.896931    2600 node_register.go:64] Registration Server started at: /registration/driver.longhorn.io-reg.sock
I0507 11:29:17.897642    2600 node_register.go:88] Skipping HTTP server because endpoint is set to: ""
I0507 11:29:18.271532    2600 main.go:90] Received GetInfo call: &InfoRequest{}
I0507 11:29:18.300677    2600 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
I0509 07:43:01.766230    2600 main.go:90] Received GetInfo call: &InfoRequest{}
I0509 07:43:01.960085    2600 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
I0513 00:27:49.210537    2600 main.go:90] Received GetInfo call: &InfoRequest{}
I0513 00:27:49.453623    2600 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
root@vm1:~# kubectl logs -f -n longhorn-system    longhorn-csi-plugin-tzjvh -p
Defaulted container "node-driver-registrar" out of: node-driver-registrar, longhorn-liveness-probe, longhorn-csi-plugin
I0507 11:25:54.267694    2364 main.go:135] Version: v2.9.2
I0507 11:25:54.267857    2364 main.go:136] Running node-driver-registrar in mode=
I0507 11:25:54.267866    2364 main.go:157] Attempting to open a gRPC connection with: "/csi/csi.sock"
W0507 11:26:04.268924    2364 connection.go:232] Still connecting to unix:///csi/csi.sock
I0507 11:26:09.634115    2364 main.go:164] Calling CSI driver to discover driver name
I0507 11:26:09.648730    2364 main.go:173] CSI driver name: "driver.longhorn.io"
I0507 11:26:09.648807    2364 node_register.go:55] Starting Registration Server at: /registration/driver.longhorn.io-reg.sock
I0507 11:26:09.649291    2364 node_register.go:64] Registration Server started at: /registration/driver.longhorn.io-reg.sock
I0507 11:26:09.649825    2364 node_register.go:88] Skipping HTTP server because endpoint is set to: ""
I0507 11:26:09.998656    2364 main.go:90] Received GetInfo call: &InfoRequest{}
I0507 11:26:10.033925    2364 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
I0509 07:04:43.727781    2364 main.go:90] Received GetInfo call: &InfoRequest{}
I0509 07:04:43.811094    2364 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
I0513 00:15:47.264722    2364 main.go:90] Received GetInfo call: &InfoRequest{}
I0513 00:15:47.388192    2364 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
I0513 00:22:07.872398    2364 main.go:90] Received GetInfo call: &InfoRequest{}
I0513 00:22:07.936918    2364 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
root@vm1:~# ^C
root@vm1:~# kubectl logs -f -n longhorn-system   longhorn-csi-plugin-x72vc  -p
Defaulted container "node-driver-registrar" out of: node-driver-registrar, longhorn-liveness-probe, longhorn-csi-plugin
I0507 11:21:35.318065    9444 main.go:135] Version: v2.9.2
I0507 11:21:35.318292    9444 main.go:136] Running node-driver-registrar in mode=
I0507 11:21:35.318301    9444 main.go:157] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0507 11:21:40.535349    9444 main.go:164] Calling CSI driver to discover driver name
I0507 11:21:40.542686    9444 main.go:173] CSI driver name: "driver.longhorn.io"
I0507 11:21:40.542757    9444 node_register.go:55] Starting Registration Server at: /registration/driver.longhorn.io-reg.sock
I0507 11:21:40.543574    9444 node_register.go:64] Registration Server started at: /registration/driver.longhorn.io-reg.sock
I0507 11:21:40.543863    9444 node_register.go:88] Skipping HTTP server because endpoint is set to: ""
I0507 11:21:41.439532    9444 main.go:90] Received GetInfo call: &InfoRequest{}
I0507 11:21:41.485203    9444 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
I0508 23:05:23.108195    9444 main.go:90] Received GetInfo call: &InfoRequest{}
I0508 23:05:23.161567    9444 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
I0510 03:01:57.858315    9444 main.go:90] Received GetInfo call: &InfoRequest{}
I0510 03:01:57.914371    9444 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
I0513 00:15:10.797240    9444 main.go:90] Received GetInfo call: &InfoRequest{}
I0513 00:15:10.830147    9444 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
I0513 00:22:07.001188    9444 main.go:90] Received GetInfo call: &InfoRequest{}
I0513 00:22:07.016034    9444 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
root@vm1:~# 

longhorn-manager-4zhvx.txt
longhorn-manager-b457p.txt
longhorn-manager-xsqpb.txt
supportbundle_b7f5b89b-7a71-41fb-aff3-0a148037c8c1_2024-05-28T15-31-52Z.zip

[ambujjain]

@PhanLe1010 Any clue with updated support bundled ?

[ambujjain]

Can someone help us here?