Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[native] Delta table access via Symlink table using Hive Connector #22580

Open
agrawalreetika opened this issue Apr 22, 2024 · 4 comments
Open
Assignees

Comments

@agrawalreetika
Copy link
Member

Delta table access via Symlink table using Hive Connector in Prestissimo

Expected Behavior or Use Case

Delta table can be accessed using hive connector by registering them as a symlink tables in HMS. https://docs.delta.io/latest/presto-integration.html
This is to extend this same support for Prestissimo using Hive Connector.

Presto Component, Service, or Connector

presto-native-execution

Possible Implementation

Currenlty mapping for inputformat and serde are missing in Prestissimo to access symlink tables. We need to add that mapping in PrestoToVeloxConnector.cpp to successfully access Delta tables.

Context

Delta table can be accessed using hive connector by registering them as a symlink tables in HMS. https://docs.delta.io/latest/presto-integration.html
This is to extend this same support for Prestissimo using Hive Connector.

@agrawalreetika
Copy link
Member Author

CC @majetideepak Fyi

@aditi-pandit
Copy link
Contributor

@tdcmeehan, @ZacBlanco

@agrawalreetika : Do you have a prototype PR ?

@agrawalreetika
Copy link
Member Author

agrawalreetika commented Apr 22, 2024

@aditi-pandit We need just mapping of Symlink input format & serde in PrestoToVeloxConnector like agrawalreetika@8760651

Discussed about this with Deepak, he mentioned we should add test for this.
So I am open to get pointers around how do we write test for this? Since Presto hive connector Java doesn't support Creating symlink file.

@majetideepak
Copy link
Collaborator

majetideepak commented Apr 22, 2024

@aditi-pandit This is similar to the Hudi change #19620.
Reetika and I are looking into how we can test Delta similar to Hudi and add a generic testing framework for both Hudi and Delta in Prestissimo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants