Method C_OpenSession returned CKR_CRYPTOKI_NOT_INITIALIZED

[lsadehaan]

I'm using PKCS11Interop v5.1.2 in a long running dotnet core 6 windows service connected to a remote Safenet HSM using the networking interface adaptor from Safenet.
Whenever the service has not been used for a while and I try opening a new session to start a new key operation I get the exception with "Method C_OpenSession returned CKR_CRYPTOKI_NOT_INITIALIZED"
Any ideas on why this happens (I'm guessing it might be within the safenet pkcs11 provider code?) and what I should do to handle this problem gracefully?
Note that it takes 5 minutes for this error response to actually occur, for 5 minutes the application just hangs waiting for a reply.
A restart of the entire application fixes the problem.