Getting Faulted state #7304

CC007 · 2023-12-08T23:49:28Z

CC007
Dec 8, 2023

Setting
I run a k8s cluster on Raspberry Pïs. I have a NAS that I hooked up to a couple of my k8s nodes. Each of the 4 drives in my NAS has its own storage volume and each of these volumes is individually accessible using NFS. 4 of my Raspberry Pis each have access to one of these volumes. I mounted them and gave the mount 755 permissions with root:root as the owner, same as /var/lib/longhorn that I was using before I added the NAS. I added each of these volume mounts as disks, using the Longhorn UI. All these disks and the nodes that they are on show as Ready and Schedulable.

Issue
When I try to create a persistent volume claim (by installing the rancher monitoring chart with persistent storage enabled for prometheus), the created volume ends up in a Faulted state, with all replicas showing Stopped. When trying to salvage, the replicas start and immediately stop again. I pulled the logs from one of the longhorn-instance-managers (one of the ones listed by the replicas) and got this:

[longhorn-instance-manager] time="2023-12-06T00:27:33Z" level=info msg="Creating instance" func="instance.(*Server).InstanceCreate" file="instance.go:87" backendStoreDriver=v1 name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
2023-12-06T01:27:33.938591926+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:33Z" level=info msg="Creating process" func="client.(*ProcessManagerClient).ProcessCreate" file="process_manager.go:83" args="[--volume-name pvc-62a32aaa-e5bd-4756-a547-5e6241832c01 replica /host/mnt/PiRackData4/replicas/pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-7758cc40 --size 53687091200 --replica-instance-name pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 --sync-agent-port-count 7]" binary=/host/var/lib/longhorn/engine-binaries/rancher-mirrored-longhornio-longhorn-engine-v1.5.3/longhorn name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 portArgs="[--listen,0.0.0.0:]" portCount=10
[longhorn-instance-manager] time="2023-12-06T00:27:33Z" level=info msg="Process Manager: prepare to create process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessCreate" file="process_manager.go:159"
[longhorn-instance-manager] time="2023-12-06T00:27:33Z" level=debug msg="Process Manager: validate process path: /host/var/lib/longhorn/engine-binaries/rancher-mirrored-longhornio-longhorn-engine-v1.5.3/longhorn dir: /host/var/lib/longhorn/engine-binaries/ image: rancher-mirrored-longhornio-longhorn-engine-v1.5.3 binary: longhorn" func=process.ensureValidProcessPath file="process_manager.go:140"
2023-12-06T01:27:33.941254779+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:33Z" level=info msg="Process Manager: created process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessCreate" file="process_manager.go:200"
2023-12-06T01:27:33.975935601+01:00 [pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578] time="2023-12-06T00:27:33Z" level=info msg="Creating replica /host/mnt/PiRackData4/replicas/pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-7758cc40, size 53687091200/512" func="replica.(*Server).Create" file="server.go:52"
[pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578] time="2023-12-06T00:27:33Z" level=warning msg="Recovering volume metafile /host/mnt/PiRackData4/replicas/pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-7758cc40/volume.meta, and replica info: {Size:53687091200 Head: Dirty:false Rebuilding:false Error: Parent: SectorSize:512 BackingFilePath: BackingFile:<nil>}" func="replica.(*Replica).tryRecoverVolumeMetaFile" file="replica.go:1088"
[pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578] time="2023-12-06T00:27:33Z" level=info msg="Starting to create disk" func="replica.(*Replica).createDisk" file="replica.go:863" disk=000
2023-12-06T01:27:34.007844128+01:00 [pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578] time="2023-12-06T00:27:34Z" level=info msg="Finished creating disk" func="replica.(*Replica).createDisk" file="replica.go:961" disk=000
[pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578] time="2023-12-06T00:27:34Z" level=fatal msg="Error running start replica command" func=cmd.ReplicaCmd.func1 file="replica.go:79" error="file extent is unsupported: operation not supported"
[longhorn-instance-manager] time="2023-12-06T00:27:34Z" level=info msg="Process Manager: process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 error out, error msg: exit status 1" func="process.(*Process).Start.func1" file="process.go:68"
[longhorn-instance-manager] time="2023-12-06T00:27:34Z" level=warning msg="Process update: pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578: state error: errorMsg: exit status 1" func="process.(*Process).RPCResponse" file="process.go:113"
[longhorn-instance-manager] time="2023-12-06T00:27:34Z" level=info msg="Stop waiting for gRPC service of process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 to start at localhost:10000" func="process.(*GRPCHealthChecker).WaitForRunning" file="healthchecker.go:29"
2023-12-06T01:27:34.935002977+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:34Z" level=warning msg="Process update: pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578: state error: errorMsg: exit status 1" func="process.(*Process).RPCResponse" file="process.go:113"
2023-12-06T01:27:35.031007592+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=warning msg="Process update: pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578: state error: errorMsg: exit status 1" func="process.(*Process).RPCResponse" file="process.go:113"
2023-12-06T01:27:35.041631428+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Getting instance log" func="instance.(*Server).InstanceLog" file="instance.go:373" backendStoreDriver=v1 name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
[longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: start getting logs for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessLog" file="process_manager.go:317"
2023-12-06T01:27:35.043884712+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: got logs for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessLog" file="process_manager.go:334"
2023-12-06T01:27:35.091553949+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Getting instance log" func="instance.(*Server).InstanceLog" file="instance.go:373" backendStoreDriver=v1 name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
2023-12-06T01:27:35.093626736+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: start getting logs for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessLog" file="process_manager.go:317"
2023-12-06T01:27:35.093990286+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: got logs for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessLog" file="process_manager.go:334"
2023-12-06T01:27:35.287208813+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Getting instance log" func="instance.(*Server).InstanceLog" file="instance.go:373" backendStoreDriver=v1 name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
2023-12-06T01:27:35.289002196+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: start getting logs for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessLog" file="process_manager.go:317"
2023-12-06T01:27:35.289259581+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: got logs for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessLog" file="process_manager.go:334"
2023-12-06T01:27:35.300498742+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Deleting instance" func="instance.(*Server).InstanceDelete" file="instance.go:149" backendStoreDriver=v1 cleanupRequired=true diskUuid=ebd33361-88a0-442e-b878-06962cc541fe name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
2023-12-06T01:27:35.302317088+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: prepare to delete process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessDelete" file="process_manager.go:209"
2023-12-06T01:27:35.302388031+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=warning msg="Process update: pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578: state error: errorMsg: exit status 1" func="process.(*Process).RPCResponse" file="process.go:113"
2023-12-06T01:27:35.302408198+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: deleted process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessDelete" file="process_manager.go:223"
2023-12-06T01:27:35.302571659+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: successfully unregistered process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).unregisterProcess.func1" file="process_manager.go:280"
2023-12-06T01:27:35.302627750+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=warning msg="Process update: pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578: state error: errorMsg: exit status 1" func="process.(*Process).RPCResponse" file="process.go:113"
[longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Getting instance log" func="instance.(*Server).InstanceLog" file="instance.go:373" backendStoreDriver=v1 name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
2023-12-06T01:27:35.315333077+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: start getting logs for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessLog" file="process_manager.go:317"
2023-12-06T01:27:35.316115029+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=error msg="Failed to receive log" func="instance.(*Server).processInstanceLog" file="instance.go:401" error="rpc error: code = NotFound desc = cannot find process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578"
2023-12-06T01:27:35.455941729+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Deleting instance" func="instance.(*Server).InstanceDelete" file="instance.go:149" backendStoreDriver=v1 cleanupRequired=true diskUuid=ebd33361-88a0-442e-b878-06962cc541fe name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
2023-12-06T01:27:35.458249457+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: prepare to delete process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessDelete" file="process_manager.go:209"
[longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Getting instance log" func="instance.(*Server).InstanceLog" file="instance.go:373" backendStoreDriver=v1 name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
2023-12-06T01:27:35.472887591+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: start getting logs for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessLog" file="process_manager.go:317"
2023-12-06T01:27:35.473305270+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=error msg="Failed to receive log" func="instance.(*Server).processInstanceLog" file="instance.go:401" error="rpc error: code = NotFound desc = cannot find process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578"
[longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Deleting instance" func="instance.(*Server).InstanceDelete" file="instance.go:149" backendStoreDriver=v1 cleanupRequired=true diskUuid=ebd33361-88a0-442e-b878-06962cc541fe name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
2023-12-06T01:27:35.628460946+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: prepare to delete process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessDelete" file="process_manager.go:209"
2023-12-06T01:27:35.639577054+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Getting instance log" func="instance.(*Server).InstanceLog" file="instance.go:373" backendStoreDriver=v1 name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
2023-12-06T01:27:35.641464898+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=info msg="Process Manager: start getting logs for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessLog" file="process_manager.go:317"
2023-12-06T01:27:35.641802301+01:00 [longhorn-instance-manager] time="2023-12-06T00:27:35Z" level=error msg="Failed to receive log" func="instance.(*Server).processInstanceLog" file="instance.go:401" error="rpc error: code = NotFound desc = cannot find process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578"
2023-12-06T01:42:11.338673030+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:11Z" level=info msg="Creating instance" func="instance.(*Server).InstanceCreate" file="instance.go:87" backendStoreDriver=v1 name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
[longhorn-instance-manager] time="2023-12-06T00:42:11Z" level=info msg="Creating process" func="client.(*ProcessManagerClient).ProcessCreate" file="process_manager.go:83" args="[--volume-name pvc-62a32aaa-e5bd-4756-a547-5e6241832c01 replica /host/mnt/PiRackData4/replicas/pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-7758cc40 --size 53687091200 --replica-instance-name pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 --sync-agent-port-count 7]" binary=/host/var/lib/longhorn/engine-binaries/rancher-mirrored-longhornio-longhorn-engine-v1.5.3/longhorn name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 portArgs="[--listen,0.0.0.0:]" portCount=10
2023-12-06T01:42:11.340687873+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:11Z" level=info msg="Process Manager: prepare to create process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessCreate" file="process_manager.go:159"
2023-12-06T01:42:11.340990035+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:11Z" level=debug msg="Process Manager: validate process path: /host/var/lib/longhorn/engine-binaries/rancher-mirrored-longhornio-longhorn-engine-v1.5.3/longhorn dir: /host/var/lib/longhorn/engine-binaries/ image: rancher-mirrored-longhornio-longhorn-engine-v1.5.3 binary: longhorn" func=process.ensureValidProcessPath file="process_manager.go:140"
2023-12-06T01:42:11.341738414+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:11Z" level=info msg="Process Manager: created process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessCreate" file="process_manager.go:200"
2023-12-06T01:42:11.374166895+01:00 [pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578] time="2023-12-06T00:42:11Z" level=info msg="Listening on data server 0.0.0.0:10001" func=cmd.startReplica.func2 file="replica.go:145"
[pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578] time="2023-12-06T00:42:11Z" level=info msg="Listening on sync agent server 0.0.0.0:10002" func=cmd.startReplica.func3 file="replica.go:174"
2023-12-06T01:42:11.419836455+01:00 time="2023-12-06T00:42:11Z" level=info msg="Listening on gRPC Replica server 0.0.0.0:10000" func=cmd.startReplica.func1 file="replica.go:137"
[pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578] time="2023-12-06T00:42:11Z" level=info msg="Listening on sync 0.0.0.0:10002" func=cmd.startSyncAgent file="sync_agent.go:89"
2023-12-06T01:42:11.468944393+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:11Z" level=info msg="Process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 has started at localhost:10000" func="process.(*GRPCHealthChecker).WaitForRunning" file="healthchecker.go:33"
2023-12-06T01:42:12.421086416+01:00 [pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578] time="2023-12-06T00:42:12Z" level=info msg="New connection from: 10.42.4.30:37550" func="rpc.(*DataServer).listenAndServeTCP" file="dataserver.go:57"
[pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578] time="2023-12-06T00:42:12Z" level=info msg="Opening replica: dir /host/mnt/PiRackData4/replicas/pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-7758cc40, size 53687091200, sector size 512" func="replica.(*Server).Open" file="server.go:72"
2023-12-06T01:42:13.291874833+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Getting instance log" func="instance.(*Server).InstanceLog" file="instance.go:373" backendStoreDriver=v1 name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
[longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Process Manager: start getting logs for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessLog" file="process_manager.go:317"
2023-12-06T01:42:13.294199246+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Process Manager: got logs for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessLog" file="process_manager.go:334"
2023-12-06T01:42:13.305404240+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Deleting instance" func="instance.(*Server).InstanceDelete" file="instance.go:149" backendStoreDriver=v1 cleanupRequired=true diskUuid=ebd33361-88a0-442e-b878-06962cc541fe name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
[longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Process Manager: prepare to delete process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessDelete" file="process_manager.go:209"
2023-12-06T01:42:13.307720949+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Process Manager: deleted process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessDelete" file="process_manager.go:223"
2023-12-06T01:42:13.307742356+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Process Manager: trying to stop process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Process).StopWithSignal.func1" file="process.go:168"
[longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Wait for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 to shutdown" func="process.(*Process).StopWithSignal.func1" file="process.go:174"
[longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=debug msg="Process Manager: wait for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 to shutdown before unregistering process" func="process.(*Manager).unregisterProcess.func1" file="process_manager.go:260"
2023-12-06T01:42:13.309450296+01:00 [pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578] time="2023-12-06T00:42:13Z" level=warning msg="Received signal interrupt to shutdown" func=cmd.registerShutdown.func1 file="shutdown.go:31"
2023-12-06T01:42:13.309521554+01:00 time="2023-12-06T00:42:13Z" level=warning msg="Starting to execute registered shutdown func github.com/longhorn/longhorn-engine/app/cmd.startReplica.func4" func=cmd.registerShutdown.func1 file="shutdown.go:34"
2023-12-06T01:42:13.344149783+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Process Manager: process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 stopped" func="process.(*Process).Start.func1" file="process.go:77"
[longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Process Manager: successfully unregistered process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).unregisterProcess.func1" file="process_manager.go:280"
2023-12-06T01:42:13.522417636+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Deleting instance" func="instance.(*Server).InstanceDelete" file="instance.go:149" backendStoreDriver=v1 cleanupRequired=true diskUuid=ebd33361-88a0-442e-b878-06962cc541fe name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
2023-12-06T01:42:13.524357091+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Process Manager: prepare to delete process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessDelete" file="process_manager.go:209"
[longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Deleting instance" func="instance.(*Server).InstanceDelete" file="instance.go:149" backendStoreDriver=v1 cleanupRequired=true diskUuid=ebd33361-88a0-442e-b878-06962cc541fe name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
2023-12-06T01:42:13.812180868+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:13Z" level=info msg="Process Manager: prepare to delete process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessDelete" file="process_manager.go:209"
2023-12-06T01:42:14.187472428+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:14Z" level=info msg="Getting instance log" func="instance.(*Server).InstanceLog" file="instance.go:373" backendStoreDriver=v1 name=pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578 type=replica
2023-12-06T01:42:14.189297496+01:00 [longhorn-instance-manager] time="2023-12-06T00:42:14Z" level=info msg="Process Manager: start getting logs for process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578" func="process.(*Manager).ProcessLog" file="process_manager.go:317"
[longhorn-instance-manager] time="2023-12-06T00:42:14Z" level=error msg="Failed to receive log" func="instance.(*Server).processInstanceLog" file="instance.go:401" error="rpc error: code = NotFound desc = cannot find process pvc-62a32aaa-e5bd-4756-a547-5e6241832c01-r-4a42a578"

Especially line 10 seems important: error="file extent is unsupported: operation not supported", but I don't know what's causing that.

I checked, and it did seem to have written data to the volume. I see files like revision.counter, volume-head-000.img, etc.

Any ideas?

mantissahz · 2023-12-12T02:23:27Z

mantissahz
Dec 12, 2023
Collaborator

Especially line 10 seems important: error="file extent is unsupported: operation not supported", but I don't know what's causing that.

Longhorn will call FIEMAP ioctl to check if the extent mapping is supported and I think FIEMAP is not supported for NFS.

4 replies

CC007 Dec 12, 2023
Author

Thanks for the info. Is there a way to configure Longhorn to get around this issue?

If not, I might also be able to expose the disks using a different method like SMB or WebDAV. Would those work or would I run into similar issues?

mantissahz Dec 13, 2023
Collaborator

CIFS should be supported by LINUX but someone just reported this https://lore.kernel.org/lkml/449658.1701866309@warthog.procyon.org.uk/T/

CC007 Dec 14, 2023
Author

From that email chain I learned (or maybe reaffirmed) that I have very limited knowledge of file systems and how they work.

I understand that there's a lot of complexity under the hood, but I'd like to keep that complexity out of sight and just go with a solution that's most commonly used.

CC007 Dec 16, 2023
Author

I switched to SMB/CIFS and (after formatting because of locking issues on the volume) got it working.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Getting Faulted state #7304

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Getting Faulted state #7304

CC007 Dec 8, 2023

Replies: 1 comment · 4 replies

mantissahz Dec 12, 2023 Collaborator

CC007 Dec 12, 2023 Author

mantissahz Dec 13, 2023 Collaborator

CC007 Dec 14, 2023 Author

CC007 Dec 16, 2023 Author

CC007
Dec 8, 2023

Replies: 1 comment 4 replies

mantissahz
Dec 12, 2023
Collaborator

CC007 Dec 12, 2023
Author

mantissahz Dec 13, 2023
Collaborator

CC007 Dec 14, 2023
Author

CC007 Dec 16, 2023
Author