Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement bulk key fetching and deleting for large queue without transaction #45

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

ota42y
Copy link

@ota42y ota42y commented Feb 18, 2022

Original PR #44

We want to reduce redis load by delete-queue command when the queue has a large number of tasks.
*a large number is about 10^6

I implemented following improvements:

  • add --without-transaction option to large queue delete

    • if the user doesn't set this option, keep old feature.
    • if the queue have many tasks and without this option, raise error.
  • use SScan instead of SMembers command to fetch task UIDs

  • use UnLink instead of Del command and bulk deleting keys by chunk size

  • WIP points

    • Two pieces of code that are doing almost the same thing.
      • with / without transaction
    • Testing the corner case
      • i.e. abort and resume deleting
    • Performance test
      • I checked this PR work well for over 1234500 tasks
      • But it's take 20 minutes so it's not good for unite test.

fixes #28

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

The same problem as #25 occurs with delete-queue for large queues
2 participants