fix: Repair MREs from #823 (#1203)

* add failing tests 823

* fix mistakes

* black

* Update tests from #823 MREs

- Can't have the b.txt in first text be temp and expect it present, it
will correctly be deleted.
- Add third MRE which is targeting more of a log issue

* Add error message printed if #823 is encountered, also exit 1

The main scheduler loop terminates if there are no more runnable
jobs. This tests if there still are jobs that should be run, but that
we cannot seem to get to. If so, print an error message, print the
files affected, and exit 1. The latter is important so Snakemake can,
as hotfix, be rerun in a loop until it's actually completed all
tasks.

* Fix checkpoint+temp leads to incomplete run issue

The aggressive early deletion of temp files must ignore
checkpoints. Other rules depend on a checkpoint only via the first
output file of the checkpoint (flagged string) as a stub. All other
outputs marked as temp would be deleted (and have been before this
patch). Since those files are missing, the dependent rules will never
be runnable and the workflow ends before all targets have been built.

* Don't break untiljobs

* fix typo

Co-authored-by: Maarten-vd-Sande <maartenvandersande@hotmail.com>
Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>
Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de>

snakemake/scheduler.py

@@ -43,6 +43,11 @@ def cumsum(iterable, zero=[0]):
     "Exiting because a job execution failed. " "Look above for error message"
 )
 
+_ERROR_MSG_ISSUE_823 = (
+    "BUG: Out of jobs ready to be started, but not all files built yet."
+    " Please check https://github.com/snakemake/snakemake/issues/823 for more information."
+)
+
 
 class DummyRateLimiter(ContextDecorator):
     def __enter__(self):
@@ -470,11 +475,23 @@ def schedule(self):
                         return False
                     continue
 
-                # normal shutdown because all jobs have been finished
+                # all runnable jobs have finished, normal shutdown
                 if not needrun and (not running or self.workflow.immediate_submit):
                     self._executor.shutdown()
                     if errors:
                         logger.error(_ERROR_MSG_FINAL)
+                    # we still have unfinished jobs. this is not good. direct
+                    # user to github issue
+                    if self.remaining_jobs:
+                        logger.error(_ERROR_MSG_ISSUE_823)
+                        logger.error(
+                            "Remaining jobs:\n"
+                            + "\n".join(
+                                " - " + str(job) + ": " + ", ".join(job.output)
+                                for job in self.remaining_jobs
+                            )
+                        )
+                        return False
                     return not errors
 
                 # continue if no new job needs to be executed

tests/test_issue823_3/Snakefile

@@ -0,0 +1,9 @@
+rule all:
+    input:
+        "a.txt"
+
+checkpoint a:
+    output:
+        "a.txt"
+    shell:
+        "touch {output}"

tests/tests.py

@@ -887,6 +887,11 @@ def test_issue823_2():
     run(dpath("test_issue823_2"))
 
 
+@skip_on_windows
+def test_issue823_3():
+    run(dpath("test_issue823_3"))
+
+
 @skip_on_windows
 def test_pathlib():
     run(dpath("test_pathlib"))