Skip to content

Commit

Permalink
fix: added missing input files in reason.updated_input in dag.py
Browse files Browse the repository at this point in the history
Let's consider a 'A' rule that takes 'N' inputs from a 'B' rule with
different wildcards. The input function for the A rule requests output
files from 'B'.

If a first run has already generated the output of 'A' and 'B' and if
the input function of 'A' requests new input files from B not yet
generated, snakemake will neither generate missing 'B' output files
nor regenerate the 'A' output. However the list-input-changes is able
to list correctly the missing files.

The commit allows to generate missing 'B' outputs and regenerate the
'A' output.
  • Loading branch information
Christophe Clienti committed Mar 7, 2022
1 parent ef0475c commit 326bf7c
Show file tree
Hide file tree
Showing 6 changed files with 23 additions and 1 deletion.
4 changes: 3 additions & 1 deletion snakemake/dag.py
Expand Up @@ -996,7 +996,9 @@ def update_needrun(job):
output_mintime_ = output_mintime.get(job)
if output_mintime_:
updated_input = [
f for f in job.input if f.exists and f.is_newer(output_mintime_)
f
for f in job.input
if (f.exists and f.is_newer(output_mintime_)) or (not f.exists)
]
reason.updated_input.update(updated_input)
if noinitreason and reason:
Expand Down
14 changes: 14 additions & 0 deletions tests/test_update_input/Snakefile
@@ -0,0 +1,14 @@
rule all:
input:
lambda wildcards: [rules.B.output[0].format(name=name)
for name in config.get("names", "john").split(",")]
output:
"A.txt"

run:
f = open(output[0], "w")
f.write(' '.join(input) + "\n")

rule B:
output:
touch("B-{name}.txt")
1 change: 1 addition & 0 deletions tests/test_update_input/expected-results/A.txt
@@ -0,0 +1 @@
B-john.txt B-doe.txt
Empty file.
Empty file.
5 changes: 5 additions & 0 deletions tests/tests.py
Expand Up @@ -1523,3 +1523,8 @@ def test_groupid_expand_cluster():
@skip_on_windows
def test_service_jobs():
run(dpath("test_service_jobs"), check_md5=False)


def test_update_input():
run(dpath("test_update_input"), cleanup=False, check_results=False)
run(dpath("test_update_input"), config={"names": "john,doe"})

0 comments on commit 326bf7c

Please sign in to comment.