Longhorn volume failed to attach

[mihiruk]

following errors are thrown When adding a Longhorn volume to a deployment and starting the deployment. But the health of the.But on the Longhorn side, the volume situation appears to be healthy. The deployment is also not active.

Rancher version :- 2.6.3
Longhorn version :- v100.2.3+up1.3.2

Error :-

MountVolume.MountDevice failed for volume "pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a" : rpc error: code = Aborted desc = volume pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a share not yet available
Unable to attach or mount volumes: unmounted volumes=[vol-wn759 kube-api-access-bsk9v], unattached volumes=[vol-wn759 kube-api-access-bsk9v]: timed out waiting for the condition

The share-manager-pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a pod in the longhorn-system namespace also restarts with the following errors.

Error :-
MountVolume.SetUp failed for volume "kube-api-access-s9m7w" : failed to fetch token: Operation cannot be fulfilled on Pod "share-manager-pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a": the UID in the bound object reference (143411ab-1125-42a8-a887-21b3ce4cef62) does not match the UID in record. The object might have been deleted and then recreated

Can you please help to solve this?

Thanks

[derekbit]

Can you provide a support bundle?

[mihiruk]

I downloaded the support bundle, shall i share it with you?

[derekbit]

Sure. Please send to longhorn-support-bundle@suse.com.

cc @james-munson for the community coordinator in this sprint.

[mihiruk]

Ok , please check the following logs as well
Thank you so much.

time="2024-01-11T03:49:33Z" level=info msg="starting RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576"
time="2024-01-11T03:49:33Z" level=info msg="ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576"
time="2024-01-11T03:49:34Z" level=debug msg="volume pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a device /dev/longhorn/pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a contains filesystem of format ext4" encrypted=false volume=pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a
time="2024-01-11T03:49:34Z" level=warning msg="failed to mount volume" encrypted=false error="'fsck' found errors on device /dev/longhorn/pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a but could not correct them: fsck from util-linux 2.36.2\n/dev/longhorn/pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a contains a file system with errors, check forced.\n/dev/longhorn/pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a: Invalid inode number for '.' in directory inode 30886.\n\n\n/dev/longhorn/pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.\n\t(i.e., without -a or -p options)\n" volume=pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a

[james-munson]

The same procedure as before should again clear the filesystem corruption and make it operational again. We'll take a look at the support bundle to see whether there is any suggestion of what might have led to this.

[james-munson]

Nothing jumps out in the logs, although that is hard to say definitively. Grepping for "level=error" along with pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a gives more than 92,000 lines.

The volume name is the same as the previous incident. In volumes.yaml, the entry for the volume shows:

  objectmeta:
    name: minio-06-1-2024
    generatename: ""
    namespace: longhorn-system
    selflink: ""
    uid: a1898e33-21b7-4483-9c16-107f8eda11bf
    resourceversion: "186783640"
    generation: 5
    creationtimestamp: "2024-01-11T06:53:33Z"
    deletiontimestamp: null
    deletiongraceperiodseconds: null
    labels:
      backup-volume: pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a
      longhornvolume: minio-06-1-2024
      recurring-job-group.longhorn.io/default: enabled
    annotations: {}
    ownerreferences: []
    finalizers:
      - longhorn.io
    clustername: ""
    managedfields:
      - manager: longhorn-manager
        operation: Update
        apiversion: longhorn.io/v1beta2
        time: "2024-01-11T06:53:33Z"
        fieldstype: FieldsV1
        fieldsv1: stripped by yq
        subresource: ""
  spec:
    size: 21474836480
    frontend: blockdev
    frombackup: nfs://192.168.189.176:/forms/longhorn-backup?backup=backup-5dbedf3b17ab49f6&volume=pvc-3ca6f4fa-a75c-4a1d-b105-1d920e5aee7a
...

Is that the problematic volume (i.e. the corruption may have been pull from the backup)? Or is it a restore done in lieu of the fsck to fix the corruption?