Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Documentation for configuring Aleph workers is lacking #3384

Open
stchris opened this issue Oct 5, 2023 · 2 comments
Open

BUG: Documentation for configuring Aleph workers is lacking #3384

stchris opened this issue Oct 5, 2023 · 2 comments
Labels
bug Things that should work, but don’t docs Issues related to Aleph’s documentation Major issue that requires attention

Comments

@stchris
Copy link
Contributor

stchris commented Oct 5, 2023

Describe the bug
It is not immediately obvious that aleph and ingest-file workers are using "number of cpus" threads, which can cause scaling issues as seen in numerous reports, the last of which is #3383 .

To Reproduce
Steps to reproduce the behavior:

  1. Follow the installation / deployment docs
  2. Massively scale up ingest-file, workers etc.
  3. Run out of postgres connections

Expected behavior
Have better control over the number of workers running. Make it visible that we are using threads by default. Explain what the servicelayer workers default to and how to disable that.

Aleph version
3.15.1

Screenshots

Additional context

@stchris stchris added bug Things that should work, but don’t docs Issues related to Aleph’s documentation triage These issues need to be reviewed by the Aleph team labels Oct 5, 2023
@tillprochaska
Copy link
Contributor

Related #3064

@Rosencrantz Rosencrantz added Major issue that requires attention and removed triage These issues need to be reviewed by the Aleph team labels Oct 24, 2023
@Rosencrantz
Copy link
Contributor

Would be good to set the limit to 0, and disable threads entirely. Also update the documentation to talk more about how we (don't) use this feature and what we recommend other people do.

stchris added a commit that referenced this issue Nov 8, 2023
simonwoerpel pushed a commit to investigativedata/aleph that referenced this issue Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Things that should work, but don’t docs Issues related to Aleph’s documentation Major issue that requires attention
Projects
None yet
Development

No branches or pull requests

3 participants