Read Only file system Not able to write into the mounted volume

[omgupta240909]

Hi Team ,
we are using longhorn for our airflow deployments and we have total 5 intances of airflow , and we have k8s cluster of 3 master nodes and 9 worker node , and we have added 1 , 1Tb of disk to all of the worker nodes. all are working fine.

suddenly we observed that few airflow instances are gone into the read only , and not able to write into it, you can see the logs below.

After troubleshooting we found that in 2 worker nodes the iscsi servise is not running so we have disable the volume scheduing into that.

after restarting the iscsi service its not coming up.

we have manually started the aliflow instances, what is reason that it goes into the read only and how do i start my iscsi service so that a will enable the longhorn volume scheduling .

please help.

[derekbit]

Your iSCSI service seems problematic and leads to IO errors in Longhorn volumes.
Can you check logs of the dmesg and iSCSI service ?

[omgupta240909]

Hi @derekbit
not able to start ISCSI service on the nodes,

[derekbit]

Hello @omgupta240909, I mean you can check the logs I mentioned to find the root cause. I think the failed iSCSI service is not caused by Longhorn.