BUG: Out of jobs ready to be started, but not all files built yet.

[chrarnold]

The help text for this bug that smakemake prints refers to #823, but this issue has been closed already, while the bug still seems to happen for newer snakemake versions.

Snakemake version
7.8

Describe the bug
Running in cluster mode. On a shared NFS file system.

Logs
...

Error executing rule dedup_umi on cluster (jobid: 13, external: 40795051, jobscript: bla/input/.snakemake/tmp.ezr_8lcd/snakejob.dedup_umi.13.sh). For error details see the cluster log and the log files of the involved rule(s).
Job failed, going on with independent jobs.
Exiting because a job execution failed. Look above for error message
BUG: Out of jobs ready to be started, but not all files built yet. Please check https://github.com/snakemake/snakemake/issues/823 for more information.
Remaining jobs:
 - dedup_umi: bla/sample8.dedup.bam
 - dedup_umi: bla//sample6.dedup.bam
 - dedup_umi: bla//sample4.dedup.bam
 - dedup_umi: bla//sample2.dedup.bam
 - dedup_umi: bla//sample3.dedup.bam
 - all: 
 - dedup_umi: bla//sample7.dedup.bam
 - dedup_umi: bla//sample1.dedup.bam

Minimal example
Cant provide one, this happens randomly or I didnt find a pattern yet

Additional context

[christopher-schroeder]

I get the same with 7.8.0

[johanneskoester]

Thanks for reporting. This is tricky. I have no idea yet what could be the issue. A minimal example would help a lot, but I can also understand that this is probably not easy to generate. At least, by @christopher-schroeder's comment, we know that it is not a cluster issue but also happens locally.

[johanneskoester]

I guess it is related to my changes in DAG.update_needrun.

[johanneskoester]

Might be fixed with PR #1691. May I ask you to try that out?

[johanneskoester]

@chrarnold I fixed one issue like this that was only occurring with -k. Did you use that flag as well?

[chrarnold]

Yes, I always use --keep-going, so sounds likely we talk about the same issue. Will test thanks! I re-open if I observe this still in 7.8.1 and above!