--default-resources combined with partial resource specification causes TypeError

[pvandyken]

Snakemake version

7.3.5+0.g4f2258ff.dirty

Describe the bug

When running snakemake using --default-resources, but without specifying mem_mb so that the Snakemake-defined default of mem_mb=max(2*input.size_mb, 1000) kicks in, along with the --resources flag with mem_mb, if a root rule specifies a mem_mb requirement under its resources directive and a downstream rule doesn't, the downstream rule will error out with

TypeError: '>' not supported between instances of 'TBDString' and 'int'

Logs

Minimal example

# Snakefile
rule all:
    input:
        "a.out"

rule a:
    output:
        "a.out"
    resources:
        mem_mb=2000, # Note that resources are specified here, but not in all.
    shell:
        "touch {output}"

# CLI call
snakemake --default-resources --resources mem_mb=4000 -c1 all

Additional context
The last version the above code will work is v6.4.1. The first version the error appears in is v6.8.0. Weirdly, in v6.7,6,5, I get completely different errors, but perhaps those issues have been resolved?

# 6.6,7
TypeError: unsupported operand type(s) for +=: 'int' and 'str'

# 6.5
AttributeError: 'list' object has no attribute 'items'

I'm currently looking into the source if the error, although it obviously stems from the fact that, in the example above, all does not know the size of it's input, so it's mem_mb gets automatically set to "<TBD>". Obviously in previous versions that would get reset when all actually ran, but this isn't happening any more.

[pvandyken]

Here's the complete trace:

Traceback (most recent call last):
  File "/path/to/snakemake/lib/python3.9/site-packages/snakemake/__init__.py", line 722, in snakemake
    success = workflow.execute(
  File "/path/to/snakemake/lib/python3.9/site-packages/snakemake/workflow.py", line 1099, in execute
    success = self.scheduler.schedule()
  File "/path/to/snakemake/lib/python3.9/site-packages/snakemake/scheduler.py", line 484, in schedule
    run = self.job_selector(needrun)
  File "/path/to/snakemake/lib/python3.9/site-packages/snakemake/scheduler.py", line 638, in job_selector_ilp
    return self.job_selector_greedy(jobs)
  File "/path/to/snakemake/lib/python3.9/site-packages/snakemake/scheduler.py", line 823, in job_selector_greedy
    a = list(map(self.job_weight, jobs))  # resource usage of jobs
  File "/path/to/snakemake/lib/python3.9/site-packages/snakemake/scheduler.py", line 897, in job_weight
    return [
  File "/path/to/snakemake/lib/python3.9/site-packages/snakemake/scheduler.py", line 898, in <listcomp>
    self.calc_resource(name, res.get(name, 0)) for name in self.global_resources
  File "/path/to/snakemake/lib/python3.9/site-packages/snakemake/scheduler.py", line 877, in calc_resource
    if value > gres:
TypeError: '>' not supported between instances of 'TBDString' and 'int'

[pvandyken]

An immediate fix that DOES work is to modify rules.resources so that every time it's called, it checks for a "<TBD>" in rules._resources. If found, it reevaluates the resources by calling self.rule.expand_resources(...) again.

Happy to make a PR, but this may be too much the "band-aid" fix for a deeper problem. Let me know what you'd like.

EDIT:
With the recent change to always using all Default resources when in Cluster mode (introduced in #1491), this will create significant problems for cluster jobs that use mem_mb. The only current workaround would be to manually set the --default-resources manually to a constant number, at least for mem_mb. So I think fixing this is pretty high priority.

[johanneskoester]

Thanks a lot, good catch! I have fixed this in above PR. Basically the call to update the resources before scheduling came too late.