Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dynamic pool size with gnumake jobclient / jobserver #369

Open
milahu opened this issue Jun 28, 2022 · 0 comments
Open

dynamic pool size with gnumake jobclient / jobserver #369

milahu opened this issue Jun 28, 2022 · 0 comments

Comments

@milahu
Copy link

milahu commented Jun 28, 2022

use gnumake jobclient to automatically grow and shrink the pool size
to limit the overall cpu load in a build system (make, cmake, ninja, ...)

this is useful on large build servers (example: 128 cores)
but the build system runs with make -j32 to limit cpu load

by default billiard.Pool(1000) and multiprocessing.Pool(1000)
will start 1000 workers *before* demand
when a jobserver is found, pool should start only one worker, and grow the pool *on* demand

for every extra worker request a token from the jobserver.
if the jobserver is full, dont start new workers

example implementation
blink-bindgen-limit-jobs-with-jobclient.patch

this can only grow the pool on demand
ideally it should also shrink the pool when workers run idle
→ release tokens to the jobserver, so other workers can run

challenge:
release tokens on exception or sys.exit
otherwise other workers can hang forever
probably with sys.excepthook

ideally "someone" would write a better jobserver protocol
where the jobserver is actively monitoring the worker processes (could also monitor cpu load)
and when a worker crashes without releasing tokens, the jobserver assumes the tokens are released
could be based on celery or similar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant