Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Need fill group results for streaming windowing aggregation in case no events #458

Open
tingfirst opened this issue Dec 30, 2023 · 4 comments · May be fixed by #464
Open

Need fill group results for streaming windowing aggregation in case no events #458

tingfirst opened this issue Dec 30, 2023 · 4 comments · May be fixed by #464
Assignees
@yl-lisen
Labels
feature

Comments

@tingfirst
Copy link
Contributor

Use case
When there is no event during specific window, current behavior of Proton can't emit that window. It ends up a bit weird semantic results.

Describe the solution you'd like
Take reference from ClickHouse: https://clickhouse.com/blog/working-with-time-series-data-and-functions-ClickHouse

Describe alternatives you've considered

Additional context
@tingfirst
Copy link
Contributor Author

@yl-lisen @chenziliang as we talked, that's a bit urgent issue we need fix.

@gangtao fyi

@yl-lisen
Copy link
Collaborator

This is a topic that has been discussed many times:

  1. Fill the default window with default values (which may not be reasonable for some aggregate results (avg/sum/...) unless the user explicitly specifies
  2. Compare Clickhouse's syntax of order by with fill step，it's not good for our streaming window aggr query, since order by is not well supported in streaming queries

For 1) means we don't enable this behavior by default.
For 2) we need use a settings fill_default_window_for_aggr=true or support EMIT DEAFULT WINDOWS , any suggests?

@tingfirst
Copy link
Contributor Author

  1. yes it should be not enabled by default
  2. For our current stage (quick fix), either setting is okay. l prefer "fill_default_window_for_aggr=true" , which can enable emit default windows (tumbling and hopping only) even without any event, and make aggr functions return 0

Can we make a quick fix asap? I need this quite urgently .

@chenziliang
Copy link
Collaborator

chenziliang commented Dec 31, 2023 •
edited

We don't need order window to fill since we emit in window order already. Setting is a quick / hack way to implement this support. We can think about SQL clause like "EMIT WITH FILL WINDOW" to enable this feature ?

@yl-lisen yl-lisen linked a pull request Dec 31, 2023 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yl-lisen yl-lisen

Labels
feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Feature/support fill missing aggr windows timeplus-io/proton
3 participants
@chenziliang @tingfirst @yl-lisen