Lock file manipulation can stop the workflow if a transient generic "Input/output error" is ever encountered

[adamnovak]

Related to #4654, the Singularity cache mutex is likely to be on a networked filesystem, and possibly on Ceph.

When going to unlock (or lock?) a file on such a filesystem, it's possible to get an "Input/output error" (errno 5). Usually this would indicate that the hard disk is broken or some other terrible thing is happening, but on distributed filesystems it might just be packet loss or a dropped connection, and trying again might fix the problem.

We should retry attempts to interact with the lock files if we get generic IO errors.

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1541