Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking: sort key support across the system #626

Open
skyzh opened this issue Apr 14, 2022 · 3 comments
Open

Tracking: sort key support across the system #626

skyzh opened this issue Apr 14, 2022 · 3 comments

Comments

@skyzh
Copy link
Member

skyzh commented Apr 14, 2022

To make our system leverage the property that our storage provides (merge sort!), we should work on each component to make them fully understand sort keys. To get a minimum e2e test to work, this tracking issue involves the following sub-tasks:

Firstly, we should have a user interface to specify the sort mode of tables. This is covered by #625. We should refactor catalog to support composite primary keys.

Secondly, we should be able to specify such properties at the storage side, and exploit the storage property to do range scan, which is covered in #589. This is not required for a full e2e to run, but is still useful.

Then, we can write some executors to leverage the sort property. We already have merge sort join, and can have things like sort agg.

Finally, the optimizer should know how each executor produces data. We should add a "sorted" property on executors #601, so as to generate optimized plans. After that, we can add rules to leverage the sort property from the storage layer. Alternative: do some ad-hoc rewrite on order by + table scan.

@skyzh
Copy link
Member Author

skyzh commented Apr 14, 2022

cc @shmiwy will take over the issue for a month. He will work on various parts of the system to gain full sort key support.

@yyin-dev
Copy link
Contributor

Status on this?

@skyzh
Copy link
Member Author

skyzh commented Sep 28, 2022

Storage now supports range filter scan. Need add support to other layers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants