Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Be able to query Delta tables generated with the new Delta protocols #22543

Open
mblanco-denodo opened this issue Apr 17, 2024 · 0 comments
Open

Comments

@mblanco-denodo
Copy link
Contributor

With the current state of the delta connector, you cannot query tables created against delta data that have been created with recent protocol versions (reading 3, writing 7). Currently it only allows querying delta tables that have been created with support for version 1 of the reading protocol and version 2 of the writing protocol.

The error that presto reports while reading the table is:

Delta protocol version (3,7) is too new for this version of Delta Standalone Reader/Writer (1,2). Please upgrade to a newer release.' Received exception with message 'Delta protocol version (3,7) is too new for this version of Delta Standalone Reader/Writer (1,2). Please upgrade to a newer release.'

Expected Behavior or Use Case

The query should return data

Presto Component, Service, or Connector

Delta connector

Possible Implementation

The connector has to use the new API Delta-Kernel, as the Delta-Standalone is being left back:
delta-io/delta#2243
delta-io/delta#2254
However, there are certain features that will be released on the next 3.2.0 version of the Delta-Kernel library, such as the ability to move to a snapshot by ID or by timestamp (currently the library only allows you to obtain data in the latest snapshot):
https://github.com/delta-io/delta/milestone/26?closed=1

Context

As the tools that generate the delta tables (such as databricks) are updated, the new tables are already generated with the new versions of the protocols, so it would be important to have delta support that could handle this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant