[BUG]No data found in pv on node

[soilidokay]

Describe the bug

I created 1 pvc and 1 pod, connecting the pvc to /data of the pod. Next I create a file test.txt in /data of the pod. The problem is when I check "pvc-1888dd63-5c65-4272-a771-04e8bd37853e-de8302e7" on the node in /dev/longhorn (using sudo mount) it is empty or when I delete the pod and recreate it the file " test.txt" also disappeared. It took me a lot of time to find out the cause and I gave up hoping someone would help. Thanks everyone.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-volv-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: longhorn-test-pod
  namespace: default
spec:
  containers:
  - name: test-container
    image: ubuntu
    command: ["/bin/sh", "-c", "while true; do echo 'Longhorn test'; sleep 3600; done"]
    volumeMounts:
    - name: longhorn-volume
      mountPath: /data
  volumes:
  - name: longhorn-volume
    persistentVolumeClaim:
      claimName: longhorn-volv-pvc

To Reproduce

I installed longhorn with rancher, I just installed it and all the configurations I left at default.

NAME                                                READY   STATUS    RESTARTS      AGE
longhorn-ui-7988c4dcf-nbtvs                         1/1     Running   0             55m
longhorn-ui-7988c4dcf-v4j7w                         1/1     Running   0             55m
longhorn-manager-nz7nz                              1/1     Running   0             55m
longhorn-manager-c9lgm                              1/1     Running   0             55m
longhorn-manager-jhpxd                              1/1     Running   0             55m
instance-manager-cdde857189b9598682d4bc20e150df80   1/1     Running   0             55m
longhorn-manager-2cfcd                              1/1     Running   1 (55m ago)   55m
instance-manager-4b65903f4ea0086375d6b0484cbd4ab0   1/1     Running   0             55m
instance-manager-b7e0d2ae794b7c2b3acbea29f9162f97   1/1     Running   0             55m
engine-image-ei-ad735e4a-wm5tl                      1/1     Running   0             55m
engine-image-ei-ad735e4a-r8742                      1/1     Running   0             55m
engine-image-ei-ad735e4a-ms7gb                      1/1     Running   0             55m
engine-image-ei-ad735e4a-x5jd5                      1/1     Running   0             55m
instance-manager-b51e67e54daccbce316ef83702e92be9   1/1     Running   0             55m
longhorn-driver-deployer-84676db47c-ht77x           1/1     Running   2 (55m ago)   55m
csi-attacher-59486bfddb-cc8xc                       1/1     Running   0             55m
csi-attacher-59486bfddb-vfdt8                       1/1     Running   0             55m
csi-provisioner-77995678f6-dvgpd                    1/1     Running   0             55m
csi-snapshotter-5ccbb45867-pdfll                    1/1     Running   0             55m
csi-resizer-64ddf8bd7f-l28wz                        1/1     Running   0             55m
longhorn-csi-plugin-8pkwn                           3/3     Running   0             55m
csi-provisioner-77995678f6-p6mbq                    1/1     Running   0             55m
csi-snapshotter-5ccbb45867-xjnfw                    1/1     Running   0             55m
csi-provisioner-77995678f6-fh9qp                    1/1     Running   0             55m
csi-resizer-64ddf8bd7f-djm8m                        1/1     Running   0             55m
csi-attacher-59486bfddb-kzwpb                       1/1     Running   0             55m
longhorn-csi-plugin-d4d8g                           3/3     Running   0             55m
csi-resizer-64ddf8bd7f-s4ddb                        1/1     Running   0             55m
csi-snapshotter-5ccbb45867-ts64j                    1/1     Running   0             55m
longhorn-csi-plugin-2lx2n                           3/3     Running   0             55m
longhorn-csi-plugin-cbvmg                           3/3     Running   0             55m
longhorn-iscsi-installation-s26wk                   1/1     Running   0             64s
longhorn-iscsi-installation-bkpc2                   1/1     Running   0             64s
longhorn-nfs-installation-rvgpv                     1/1     Running   0             59s
longhorn-nfs-installation-z5b5v                     1/1     Running   0             59s
longhorn-iscsi-installation-7rj7l                   1/1     Running   0             64s
longhorn-iscsi-installation-59wsq                   1/1     Running   0             64s
longhorn-nfs-installation-lbdkx                     1/1     Running   0             59s
longhorn-nfs-installation-g7c95                     1/1     Running   0             59s

Expected behavior

Support bundle for troubleshooting

Environment

• Longhorn version: 1.6.1
• Impacted volume (PV):
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Rancher Catalog App
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: microk8s
  • Number of control plane nodes in the cluster: 1
  • Number of worker nodes in the cluster:3
• Node config
  • OS type and version:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

• Kernel version: 6.5.0-35-generic
• CPU per node:
• Memory per node: 100GI
• Disk type (e.g. SSD/NVMe/HDD):SSD
• Network bandwidth between the nodes (Gbps): 1000
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
• Number of Longhorn volumes in the cluster: 1

Additional context

[PhanLe1010]

@soilidokay Can I have more details? So you deploy the yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-volv-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: longhorn-test-pod
  namespace: default
spec:
  containers:
  - name: test-container
    image: ubuntu
    command: ["/bin/sh", "-c", "while true; do echo 'Longhorn test'; sleep 3600; done"]
    volumeMounts:
    - name: longhorn-volume
      mountPath: /data
  volumes:
  - name: longhorn-volume
    persistentVolumeClaim:
      claimName: longhorn-volv-pvc

Then how did you create a file test.txt in /data of the pod ?