Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf: improve job selection performance in case of potential ambiguity that is resolved by comprehensive ruleorder statements. #1147

Merged
merged 3 commits into from Sep 6, 2021
Merged

perf: improve job selection performance in case of potential ambiguity that is resolved by comprehensive ruleorder statements. #1147

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0a5c498
perf: improve job selection performance in case of potential ambiguit…
johanneskoester Aug 23, 2021
a4d840a
fmt
johanneskoester Aug 23, 2021
6138d4f
improve order checking optimization after correct remarks from @dkuzm…
johanneskoester Aug 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
11 changes: 9 additions & 2 deletions snakemake/dag.py
View file
Open in desktop
Expand Up @@ -735,6 +735,11 @@ def update(
exceptions = list()
cycles = list()

# check if all potential producers are strictly ordered
jobs = sorted(jobs, reverse=True)
primary_job = jobs[0]
is_strictly_ordered = all(primary_job > job for job in jobs)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not correct, as the first element shall be compared to the rest of the elements:

is_strictly_ordered = all(primary_job > job for job in jobs[1:])

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch!


for job in jobs:
logger.dag_debug(dict(status="candidate", job=job))
if file in job.input:
Expand All @@ -753,8 +758,10 @@ def update(
progress=progress,
create_inventory=create_inventory,
)
# TODO this might fail if a rule discarded here is needed
# elsewhere
if is_strictly_ordered:
# Jobs are strictly ordered, hence the first working one is the
# one to use and all others can be skipped.
break
producers.append(job)
except (
MissingInputException,
Expand Down