feat: trigger rerun when new files are yield by an input function

[cclienti]

Description

Added the "missing" input flag to trigger rerun when new files are yield by an input function. Such flag is forwarded to all elements yield by the input function.

QC

• The PR contains a test case for the changes or the changes are already covered by an existing test case.
• The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[johanneskoester]

Hi, thanks a lot for the proposal. However, I think Snakemake has a mechanism for that purpose already, which is --list-input-changes. The latest versions of Snakemake already warn you in such cases at the end of a dryrun and suggest a command to trigger a rerun. Hence, there is not even manual flagging needed as you have implemented it here. What do you think?

[cclienti]

Thank you for the response. I agree and I tried the proposed mechanism, but sometimes the command line length is too long because of thousands new files. Moreover the DAG is analyzed two times which can be long for some large workflows (>100k nodes).

So, in such case I was forced to create a snakemake-update python code that parses the arguments (using the internal API) to add the "list-input-changes" option and run snakemake to get the target python list. Then the snakemake can run a second times with the python list of changes. A same class solution would be to pass to snakemake command line the targets using a file, but the two run mechanism would be the same.

I was thinking that adding a flag would be nicer solution for such use case (only one snakemake run). But maybe I did not see the big picture or implemented things correctly.

[johanneskoester]

I agree about the wasting of runtime. There is anyway the plan to implement these auxilliary change detections as an arg like --rerun-changes params input code. The mechanism would be basically the same as now, but no recomputation would be needed because it can happen directly before the needrun determination in the DAG. Would that be a solution to your problem?

[johanneskoester]

See also here: #1107

So, maybe it is time to finally implement that now. I'll try to come to that ASAP.

[cclienti]

I agree about the wasting of runtime. There is anyway the plan to implement these auxilliary change detections as an arg like --rerun-changes params input code. The mechanism would be basically the same as now, but no recomputation would be needed because it can happen directly before the needrun determination in the DAG. Would that be a solution to your problem?

Yes this is also a good solution 👍

[johanneskoester]

This will now be handled differently but as desired in PR #1663. Thanks for the help!

[cclienti]

Thank you for mentioning me in the release note 7.8.0. You just made a little error on my nick, this is cclienti, not cclienty :D