Skip to content
This repository has been archived by the owner on Mar 12, 2020. It is now read-only.

Improve Sidekiq queue efficiency #1381

Open
BenEmdon opened this issue Jul 4, 2018 · 5 comments · May be fixed by #2518
Open

Improve Sidekiq queue efficiency #1381

BenEmdon opened this issue Jul 4, 2018 · 5 comments · May be fixed by #2518

Comments

@BenEmdon
Copy link
Contributor

BenEmdon commented Jul 4, 2018

Feature request ✨

Currently classroom uses sidekiq queues inefficiently.
This is our current queue setup:

:queues: # https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues
- [assignment, 1]
- [chewy, 1]
- [last_active, 1]
- [github_event, 3]
- [trash_can, 2]
- [deadline, 3]
- [boom, 3]

This is what the Sidekiq docs say about having lots of queues:

I don't recommend having more than a handful of queues. Lots of queues makes for a more complex system and Sidekiq Pro cannot reliably handle multiple queues without polling.
https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues

One way we could improve this would be to use the regular 3 queues (critical, default, and low).

@stale
Copy link

stale bot commented Sep 25, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale
Copy link

stale bot commented Dec 12, 2018

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

@stale
Copy link

stale bot commented Sep 21, 2019

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

@stale stale bot added the stale label Sep 21, 2019
@v4vasanth
Copy link

Hi! I can help with this ✋

@gustavosobral
Copy link
Contributor

I think I can do this one. Have you done any work on this @v4vasanth?


@BenEmdon So, your idea here is to just create these three queues that you've mentioned and re-assign the workers? I thought about doing something like this:

:queues: # https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues
  - [critical, 3]
  - [default, 2]
  - [low, 1]

And assigning the Workers that have a queue with priority 3 to the critical queue and so on.

Other option would be this setup:

:queues: # https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues
  - critical
  - default
  - low

But from the Sidekiq documentation we have that:

This means that any job in the default queue will be processed only when the critical queue is empty.

I don't like the idea of only processing from default when critical is totally empty. So I will go for the option one.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants