BUG: Complete log file is incomplete

[Rebecza]

Describe the bug
The (more detailed) errors thrown by the pipeline are not documented in the complete log files generated at the end of the run.

I want to trace back what the cause of the errors was, in the complete logger produced at the end. For instance, the detailed description of the rule that fails or the detailed statements in the raised CalledProcessError exception, these can not be found back.

Screenshots
An example of my error:

And this is the only information that can be found in the complete log seq2science.2021-04-22T120203.500592.log:

[siebrenf]

Right, the snakemake logger can only pickup stderr OR stdout. We could redirect the stderr with contextlib, but also need to print it to the console.

[Maarten-vd-Sande]

I am not entirely sure. But I think the piece of:

Error in rule XXX
......

piece is also missing? @Rebecza

[Maarten-vd-Sande]

I think (not sure) this is related to the rule being executed in the run: directive... 🤔

[siebrenf]

That's it!

I can recreate logs missing Error in rule... by canceling this test run:
seq2science run download-fastq --cores 2 --configfile tests/download_fastq/default_config.yaml

However, the following test run does have the Error in rule... when cancelled (or by placing an exit(1) in script get_genome.py):
seq2science run alignment --cores 2 --configfile tests/alignment/remote_genome_n_sample.yaml --snakemakeOptions until=[get_genome_annotation]

The difference is the top one uses run:, the bottom uses script:.

[Maarten-vd-Sande]

Ok that's annoying. 😞

I don't think we can fix this at the moment. Perhaps if we upgrade the snakemake version in the future this gets solved? 🤷

[Maarten-vd-Sande]

Ok perhaps if we just capture all stdout and stderr, and redirect it to the s2s logfile, we have the best of both worlds? We capture everything, including the exceptions? 🧐 🤔

[Rebecza]

I am not entirely sure. But I think the piece of:

Error in rule XXX
......

piece is also missing? @Rebecza

This probably comes to late: but yes, there is no printing of any "error"-containing-lines in the complete log, besides the very last:
Exiting because a job execution failed. Look above for error message

[Maarten-vd-Sande]

Ok I tried this in a test Snakefile. This is a snakemake bug unfortunately...

[Maarten-vd-Sande]

Filed a report here: snakemake/snakemake#1004