Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Truncate Redis Streams #1270

Closed
ddimaria opened this issue Apr 8, 2024 · 0 comments · Fixed by #1278
Closed

Truncate Redis Streams #1270

ddimaria opened this issue Apr 8, 2024 · 0 comments · Fixed by #1278
Assignees
Labels
high priority within weeks type: enhancement new feature or request

Comments

@ddimaria
Copy link
Collaborator

ddimaria commented Apr 8, 2024

Redis streams are filling up the cache with legacy transactions. In a separate thread in the Files service, truncate messages every hour (or day?) based on the following process/rules:

  • Query Redis for all active channels
  • For each active channel, query the API service for the latest checkpoint (or one prior?)
  • Truncate the stream (channel) up to the set sequence number of the checkpoint

Things to consider:

  • Optimize the lookups to only trim logs that need trimming (could require more persistence in API)
  • Consider auto-truncating after creating a new checkpoint in the Files service to avoid a separate thread potentially creating race conditions
@ddimaria ddimaria added type: enhancement new feature or request high priority within weeks labels Apr 8, 2024
@ddimaria ddimaria self-assigned this Apr 8, 2024
@ddimaria ddimaria mentioned this issue Apr 18, 2024
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
high priority within weeks type: enhancement new feature or request
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant