fix: Don't handle the same job output error twice when --keep-going is set

[cjops]

Description

Hoping I understand Snakemake's scheduler loop logic correctly. This should fix #1244.

QC

• The PR contains a test case for the changes or the changes are already covered by an existing test case.
• The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[iamh2o]

++ support to adopting this fix ASAP. I am going to pull it and give it a whirl right now b/c the seriousness of the impact on operations has reached a point workarounds are failing adI was exploring next options when happily noted this in my inbox. For more words, see here. Will report back- i can test BIG jobs running via qsub and drmaaa on uge.

[cstenkamp]

I seem to have the same issue as @iamh2o - would be great if this PR could find it's way into a Release ASAP!

[johanneskoester]

Thanks a lot! I recently applied this modification as well, in addition to some other fixes in PR #1332.

[iamh2o]

So far so good with snakemake v7.0.1 -- 5K jobs and no crashes. I'll force the failure condition when I have some down time and that will be a best test I have (it just takes a fair amount of prep as a ton of jobs need to be in flight.