Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve modularity of presto-hive-common module #22774

Merged
merged 1 commit into from
May 22, 2024

Conversation

abhiseksaikia
Copy link
Contributor

Refactoring presto-hive-common module by creating a separate module for files required by presto-parquet and presto-cache modules to reduce dependency clutter

Description

This change aims to reduce the friction between the Prism connector and the Hive connector by separating out some of the classes from presto-hive-common into a new module called presto-hdfs-core. This will allow for better decoupling between the two connectors and eliminate the need for weird runtime interfaces.

Motivation and Context

The Prism connector is where most of Meta's specific integration points are, and it takes the OSS Hive connector as a runtime dependency to reduce code duplication. However, differences in file formats, metastore, and storage create a lot of friction. The current approach of introducing unclean runtime interfaces to support Prism specific use cases creates strong coupling between the two connectors. By separating out some of the classes from presto-hive-common into presto-hdfs-core, we can eliminate the dependency on presto-hive-common from non-connector modules. This will make it easier to maintain and update the codebase, and reduce the risk of introducing bugs or issues when making changes to the connectors. Additionally, this change will make it easier to add new features or functionality to the Prism connector without affecting the Hive connector.

Test Plan

  • Unit test
  • Shadow test

Contributor checklist

  • Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.

Release Notes

== NO RELEASE NOTE ==

Refactoring presto-hive-common module by creating a separate module for
files required by presto-parquet and presto-cache modules to reduce
dependency clutter
@tdcmeehan tdcmeehan self-assigned this May 17, 2024
@abhiseksaikia abhiseksaikia changed the title [WIP] Improve modularity of presto-hive-common module Improve modularity of presto-hive-common module May 17, 2024
@abhiseksaikia abhiseksaikia marked this pull request as ready for review May 17, 2024 20:08
Copy link
Contributor

@elharo elharo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

@facebook-github-bot
Copy link
Collaborator

@abhiseksaikia has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@arhimondr arhimondr merged commit b258543 into prestodb:master May 22, 2024
57 of 59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants