Skip to content

Commit

Permalink
fix: enable usage of job grouping in GLS (#1054)
Browse files Browse the repository at this point in the history
* Allow using grouped rules on Google cloud using Google Life Science
API

* Explain new behaviour in the docs

* Modify behaviour to request all rules in a group to be set as preemptible instances

* Fix typo and add comment about documentation

Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>
  • Loading branch information
sposadac and johanneskoester committed Oct 21, 2021
1 parent 19f5a43 commit d243c22
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 1 deletion.
2 changes: 2 additions & 0 deletions docs/snakefiles/rules.rst
Expand Up @@ -1492,6 +1492,8 @@ However, if we would add ``group: "mygroup"`` to rule ``c``, all jobs would end
Alternatively, groups can be defined via the command line interface.
This enables to almost arbitrarily partition the DAG, e.g. in order to safe network traffic, see :ref:`here <job_grouping>`.

For execution on the cloud using Google Life Science API and preemptible instances, we expect all rules in the group to be homogenously set as preemptible instances (e.g., with command-line option ``--preemptible-rules``), such that a preemptible VM is requested for the execution of the group job.

Piped output
------------

Expand Down
13 changes: 12 additions & 1 deletion snakemake/executors/google_lifesciences.py
Expand Up @@ -511,12 +511,23 @@ def _generate_job_resources(self, job):
"Selected machine type {}:{}".format(smallest, selected["description"])
)

if job.is_group():
preemptible = all(rule in self.preemptible_rules for rule in job.rules)
if not preemptible and any(
rule in self.preemptible_rules for rule in job.rules
):
raise WorkflowError(
"All grouped rules should be homogenously set as preemptible rules"
"(see Defining groups for execution in snakemake documentation)"
)
else:
preemptible = job.rule.name in self.preemptible_rules
# We add the size for the image itself (10 GB) to bootDiskSizeGb
virtual_machine = {
"machineType": smallest,
"labels": {"app": "snakemake"},
"bootDiskSizeGb": disk_gb + 10,
"preemptible": job.rule.name in self.preemptible_rules,
"preemptible": preemptible,
}

# If the user wants gpus, add accelerators here
Expand Down

0 comments on commit d243c22

Please sign in to comment.