Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fan-out support for the same source #288

Open
yl-lisen opened this issue Nov 17, 2023 · 10 comments · May be fixed by #314
Open

Fan-out support for the same source #288

yl-lisen opened this issue Nov 17, 2023 · 10 comments · May be fixed by #314
Assignees
@yl-lisen
Labels
feature

Comments

@yl-lisen
Copy link
Collaborator

The multiplexer fans out one streaming store reader to different streaming queries.
This has efficiency of disk read / TFF deserialization, memory allocation etc.
But we may introduce new problems like one slow query processing pipeline will slow down other pipelines.
So it maintains some metrics for channel and detach the channel and its the pipeline when slowness is detected.

                                                            -> StreamingStoreSourceChannel -> QueryProcessing Pipeline
StreamingStore Partition -> StreamingStoreSourceMultiplexer -> StreamingStoreSourceChannel -> QueryProcessing Pipeline
                                                            -> StreamingStoreSourceChannel -> QueryProcessing Pipeline

Multiplexer is multiple-thread safe, and a multiplexer is bound to a shard
@yl-lisen yl-lisen self-assigned this Nov 17, 2023
@yl-lisen yl-lisen linked a pull request Nov 22, 2023 that will close this issue
Open
@jovezhong
Copy link
Contributor

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

9 similar comments
@jovezhong
Copy link
Contributor

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

@jovezhong
Copy link
Contributor

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

@jovezhong
Copy link
Contributor

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

@jovezhong
Copy link
Contributor

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

@jovezhong
Copy link
Contributor

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

@jovezhong
Copy link
Contributor

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

@jovezhong
Copy link
Contributor

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

@jovezhong
Copy link
Contributor

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

@jovezhong
Copy link
Contributor

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yl-lisen yl-lisen

Labels
feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Feature/enable fan out for the same source timeplus-io/proton
2 participants
@jovezhong @yl-lisen