Implement rule-specific restart_times directive

[kpj]

It is possible to call snakemake with --restart-times to automatically restart a rule if it crashes (and, e.g., increase resource requirements for the next run). So far this is only possible for all rules together and lacks specificity.

This PR introduces a rule-specific restart-times directive which allows to specify the number of restart attempts for a specific rule. It fixes #367.

A (silly) example could look like this:

rule all:
    input:
        "foo",


rule foo:
    output:
        touch("foo"),
    restart_times:
        1
    resources:
        mem_mb=lambda wildcards, attempt: 10 * attempt,
    run:
        print(resources.mem_mb)
        1 / 0

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[kpj]

At the moment there is an issue where rules which set the restart_times directive are executed too often because I do not understand how Snakemake's execution internals work.
The global variable set by the --restart-times commandline parameter is reset to 0 for rule executions and only retains its actual value the first time main() and thus exec(compile(..), ..) is executed.
I cannot figure out how this reset works and cannot apply it to the rule-specific restart times 😞

In addition, I'd like to make it possible to pass a function to restart_times which returns True if another attempt is supposed to be made and False otherwise.

I'd be happy to hear how one could solve this.

[kpj]

@johanneskoester Do you have any suggestion regarding the implementation? :-)

[kpj]

This is now reasonably resolved by #1649.