New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

start celery pool with concurrency == number of cpus * 3 #410

Open

dtenenba wants to merge 1 commit into master from grow-celery-pool

Contributor

dtenenba commented Aug 2, 2022

Since our jobs are IO-bound they are not CPU intensive and we can make the pool deeper.

For reference, pool size can be increased at runtime by docker exec'ing into the celery container and running:

source ./load-secrets.sh
celery -A api.tasks control pool_grow 20 # adds 20 to current size


          start celery pool with concurrency == number of cpus * 3

0e7adab

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment