Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support to be able to query new Delta protocols #22596

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

mblanco-denodo
Copy link
Contributor

@mblanco-denodo mblanco-denodo commented Apr 24, 2024

Description

Use of delta-kernel-api instead of delta-standalone-api in order to be able to query delta tables in newer protocols (delta reader 3, delta writer 7)
Added new configurable parameter (delta.case-sensitive-partitions-enabled) to allow case sensitive partition matching, allowing to query tables with partition names in uppercase. By default it is set to true.

Motivation and Context

Solves #22543 and #21828.

Impact

Performance impact has not been measured yet

Test Plan

Test refactor so that the current tests are all executed on older and newer versions of the same test data.

Contributor checklist

  • Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

Delta Connector Changes

  • Added support to read tables up to Delta Protocol Version 3,7 through the use of the new Delta-Kernel api
  • Added new boolean configuration parameter delta.case-sensitive-partitions-enabled to be able to query data with partitioned columns with column names in uppercase. This property is set to true by default.

@mblanco-denodo
Copy link
Contributor Author

PR set to draft while waiting for the delta kernel api to release v3.2.0. https://github.com/delta-io/delta/milestone/26
Current limitations that will be solved on 3.2.0:

  • Move to an snapshot different from the latest.
  • Move to an snapshot by timestamp
  • Reading partition columns of TimestampType is unsupported

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant