Scheduler occasionally messes up in Snakemake 5.30.1 #771
Comments
|
FWIW I have also seen this behavior on version |
|
Does anyone know if this issue was ever addressed? Because while it is less common in v5.25.0, I still see this issue occasionally. Edit: Actually, not exactly the same error. I'll put together a new issue for this. Still want to know if anyone ever figured this out. |
|
Just checked, and still seeing this issue (or something similar to it) in the most recent version of snakemake (6.4.1). I've also started seeing solver issues in v5.25.0, which is what I've been using to avoid this issue; the solver issue I'm seeing in 5.25.0 doesn't seem to be an issue in 6.4.1, but it puts me in a odd spot where I will probably have to tell my users that they can't use Mac OS X (the second solver issue doesn't seem to happen on Linux). The second solver issue seems to be independent of this issue, and results in the following showing up periodically. As best I can tell, glpsol seems to be installed on the system in question (Mac OS X Sierra). Again, this issue only shows up in v5.25.0 (which I know isn't the most recent version, but avoids the scheduler issue that originated this thread). |
So this error here should be fixed in the latest versions. We don't have the resources to backport such fixes. However, regarding your real issue here (the one reported at the top): can you reproduce that when setting |
|
Error 2 seems to have been fixed in the latest versions of Snakemake (my suspicion is that there was something wrong in the conda setup script for Mac OS X) With regards to the first: Step 1: Make a test environment running the latest version. The error still happens (although since I now use Step 2: Re-run test with Result: Still fails. I get the following error (from the scheduler; doesn't end up in the .snakemake log file): Just for good measure, the rule that keeps failing due to this is: and the rule that would generate that file is: |
|
And just to make sure, the rule BLAST is not executed for the respective sample before the failing rule? |
|
I've attached the complete logs generated by Snakemake for both the ILP and greedy solvers; the ILP solver eventually gets around to running the rule BLAST; the greedy solver doesn't and stops shortly after the error. In addition, there was some output to the terminal from the greedy solver version that didn't make it to the log file for whatever reason, so I also copied the terminal output from that version. The error in the terminal output version happens at line 4455. ILP solver: greedy solver: |
|
Do you have any checkpoints or dynamic() in that worklfow? |
|
I'm asking because that must be a quite extreme corner case. I have never seen anything like that, in hundreds of real world applications. |
|
I don't have any use of the checkpoint keyword at the moment, though I was considering trying to add them to see if that fixed anything. I'm running the test case set from https://github.com/Kennedy-Lab-UW/Duplex-Seq-Pipeline to produce this right now. |
|
Ah, wait. I think I have spotted it! The input file is marked as ancient in PostBlastProcessing1. |
|
Usually, that was intended for files that are not generated by other jobs. So the behavior you are seeing here is kind of undefined. May I ask what behavior you intend to achieve by marking the file as ancient? |
|
Of course, Snakemake should either forbid that or behave predictive in some way. Just not yet sure in what way, hence I am asking. |
|
I believe my logic when I defined it that way was that Snakemake was occasionally rerunning that step when I didn’t think it needed to. My intention was something like “If the file exists, treat it as if it does not need updating; if the file doesn’t exist, create it”. I’d sort of expect that, even if the file is marked as ancient, the pipeline would still check for existence and generate it from other rules if necessary. |
|
Yes, I agree. This makes sense. |
|
With a small example, I cannot reproduce the problem though. Could you please remove the ancient flag and see if the problem goes away? |
|
It seems to (removed |
|
Looks like others are running into this issue, too: #946 |
|
Closing as the issue is stale and appears to be resolved. Please comment if this is a mistake. |
ghost
mentioned this issue
Snakemake version
5.30.1 confirmed; does not occur in 5.25.0
Describe the bug
Occasionally, the scheduler decides to run a rule before all the input files are present. The rule then crashes, which crashes the pipeline. Restarting the pipeline sometimes fixes it, but I don't know if it will always fix it.
Logs
This is the snakemake log from the particular run. At this point, my log files don't contain anything; they're just there for when I decide to do something with them.
This pipeline crashed due to the file "Intermediate/postBlast/test2_dcs.blast.xml" not being found by the python script.
Minimal example
See test cases for https://github.com/Kennedy-Lab-UW/Duplex-Seq-Pipeline/tree/v2.0.0_prerelease; I was testing out a new version of my pipeline when I came across this issue. The test cases work just fine with snakemake=5.25.0 (at least, the pipeline runs to completion; whether the data is correct is not an issue for here). Both times I tried running them with snakemake=5.30.1, they crashed (one of those times, I tried restarting the pipeline and it ran the rest of the way through).
Additional context
I'm testing on WSL1 Ubuntu, but a colleague of mine also observed this on WSL2 Ubuntu. It doesn't seem to be related to the python version (happened with both python=3.8 and python=3.7)
The text was updated successfully, but these errors were encountered: