Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3 download concurrency (read log) #74

Open
danthegoodman1 opened this issue Aug 12, 2023 · 0 comments
Open

S3 download concurrency (read log) #74

danthegoodman1 opened this issue Aug 12, 2023 · 0 comments
Labels
enhancement New feature or request help wanted Extra attention is needed
Milestone
v3.1

Comments

@danthegoodman1
Copy link
Owner

For reading the state, if we can sequentially download the log files and pielines them so they are still processed in order. Say we try to keep a buffer of 10 in memory that are ready to be processed.
@danthegoodman1 danthegoodman1 added the enhancement New feature or request label Aug 12, 2023
@danthegoodman1 danthegoodman1 changed the title S3 download concurrency S3 download concurrency (read state) Aug 12, 2023
@danthegoodman1 danthegoodman1 added the help wanted Extra attention is needed label Aug 13, 2023
@danthegoodman1 danthegoodman1 mentioned this issue Aug 13, 2023
Closed
@danthegoodman1 danthegoodman1 changed the title S3 download concurrency (read state) S3 download concurrency (read log) Aug 13, 2023
@danthegoodman1 danthegoodman1 added this to the v3.1 milestone Aug 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Milestone
v3.1
Development

No branches or pull requests
1 participant
@danthegoodman1