fix: efficient job status checking when using DRMAA API (this should yield much better parallelization and performance when using --drmaa)

[johanneskoester]

Description

see above

QC

• The PR contains a test case for the changes or the changes are already covered by an existing test case.
• The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

[marrip]

I tried running this branch on our infrastructure and it seems something is causing a error:

Traceback (most recent call last):
  File "/home/marrip/snakemake_drmaa_test/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 727, in _wait_thread
    self._wait_for_jobs()
  File "/home/marrip/snakemake_drmaa_test/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 1458, in _wait_for_jobs
    suspended_msg.remove(active_job.job.jobid)
KeyError: 1657

[johanneskoester]

I tried running this branch on our infrastructure and it seems something is causing a error:

Traceback (most recent call last):
  File "/home/marrip/snakemake_drmaa_test/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 727, in _wait_thread
    self._wait_for_jobs()
  File "/home/marrip/snakemake_drmaa_test/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 1458, in _wait_for_jobs
    suspended_msg.remove(active_job.job.jobid)
KeyError: 1657

Thanks for trying. I think this should be fixed now. Can you give it another try?

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[marrip]

Hey Johannes,

lewking very promising. I submitted a total of 17847 jobs and atm it is running smoothly. I will keep an eye on it for a bit but it seems to work 🎉

[marrip]

I had the workflow running now for about 2 days and it is down again to 20, sometimes 40 jobs even though I see it could submit more jobs as the dependencies are met and I set a limit to 100 jobs. Yesterday, so after 1 day, it still looked fine but now the lag seems to be back...

[johanneskoester]

I had the workflow running now for about 2 days and it is down again to 20, sometimes 40 jobs even though I see it could submit more jobs as the dependencies are met and I set a limit to 100 jobs. Yesterday, so after 1 day, it still looked fine but now the lag seems to be back...

Mhm, do you get any of these messages here: https://github.com/snakemake/snakemake/pull/1156/files#diff-438f3317205fd7130727d0589d2fc1a6c2e1f6fc48c2c04d354a8a09b91ba2f4R1447?

[marrip]

I checked the logs and found 14 of them for the total of 2 days running it. I think that should not be the reason why it's lagging. But somehow the workflow recovered now and is back to 100 jobs. I am not quite sure what the problem was. Could it be related to many short small jobs that snakemake has problems "catching up" due to filesystem latency etc.?

[johanneskoester]

Yes, that makes sense. Maybe those jobs had finished, but the main process was then waiting for their output files to become visible. I am relieved to hear that is seems to work then :-). Let me merge this, but please contact me via discord if other problems occur with DRMAA. I really want this to work as good as possible.

[marrip]

Sure, I will keep you updated if I see any inconsistent behavior again. Thank you so much for your help and support!