Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: enable usage of job grouping in GLS #1054

Merged
merged 6 commits into from Oct 21, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/snakefiles/rules.rst
Expand Up @@ -1461,6 +1461,8 @@ However, if we would add ``group: "mygroup"`` to rule ``c``, all jobs would end
Alternatively, groups can be defined via the command line interface.
This enables to almost arbitrarily partition the DAG, e.g. in order to safe network traffic, see :ref:`here <job_grouping>`.

For execution on the cloud using Google Life Science API and preemptible instances, we expect all rules in the group to be homogenously set as preemptible instances (e.g., with command-line option ``--preemptible-rules``), such that a preemptible VM is requested for the execution of the group job.

Piped output
------------

Expand Down
13 changes: 12 additions & 1 deletion snakemake/executors/google_lifesciences.py
Expand Up @@ -511,12 +511,23 @@ def _generate_job_resources(self, job):
"Selected machine type {}:{}".format(smallest, selected["description"])
)

if job.is_group():
preemptible = all(rule in self.preemptible_rules for rule in job.rules)
if not preemptible and any(
rule in self.preemptible_rules for rule in job.rules
):
raise WorkflowError(
"All grouped rules should be homogenously set as preemptible rules"
"(see Defining groups for execution in snakemake documentation)"
)
else:
preemptible = job.rule.name in self.preemptible_rules
# We add the size for the image itself (10 GB) to bootDiskSizeGb
virtual_machine = {
"machineType": smallest,
"labels": {"app": "snakemake"},
"bootDiskSizeGb": disk_gb + 10,
"preemptible": job.rule.name in self.preemptible_rules,
"preemptible": preemptible,
}

# If the user wants gpus, add accelerators here
Expand Down