Add support for REST catalog in Iceberg connector #22417

kiersten-stokes · 2024-04-03T23:16:26Z

Description

This PR is two commits:

one that adds support for REST, which necessitates updating com.fasterxml.jackson.core:jackson-core and :jackson-databind to be >2.11 in the root pom
one that re-factors the Iceberg catalog implementation to be more modular (contains the bulk of the code changes)

Motivation and Context

Closes #20422

Impact

Adds support for using the Iceberg REST catalog

Test Plan

Tests have been added that spin up a testing HTTP server to create an Iceberg REST catalog with JDBC backing. Currently, three of the distributed smoke tests fail when using this set up due to a bug in Iceberg that I intend to follow up on

Contributor checklist

Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

General Changes
* Upgrades fasterxml.jackson artifacts to 2.11 :pr:`22417`

Iceberg Connector Changes
* Add support for Iceberg REST catalog :pr:`22417`

steveburnett

Thanks for the documentation! Just two nits.

presto-docs/src/main/sphinx/connector/iceberg.rst

ZacBlanco

Changes look good overall except that I think I disagree with leaving the refactor until later on IcebergResourceFactory. Might even be better to have this PR with 2 commits: one for the refactor and one for the REST catalog?

How were you thinking about refactoring it?

steveburnett

LGTM! (docs)

Pull updated branch, new local build. Everything looks good.

hantangwangd · 2024-04-06T01:44:49Z

For reference, I agree with @ZacBlanco, use a separate commit for refactoring IcebergResourceFactory in the same PR would make the code more clearer.

kiersten-stokes · 2024-04-08T00:38:01Z

@ZacBlanco @hantangwangd love the idea of doing two separate commits. That would be the best of both worlds since I can keep it separate from the functionality but also get the refactor done sooner than later

As for the re-factor, there are certainly several ways to go about it. I'm currently (at least once I'm back from vacation) considering two methods. First is passing a newSetBinder to the IcebergResourceFactory and accessing the relevant methods from there. This may be a bit easier. The second method is probably more involved, separating out each type of Iceberg catalog into it's own module and extending all the relevant classes from there (I think just config and resource factory's getCatalogProperties, though I haven't delved deep into this yet).

Please feel free to suggest additional ways and to weigh in on the above! As mentioned, I won't be back to this in another day or two 🙂

ZacBlanco · 2024-04-16T13:34:04Z

I took a brief look. Re-factor overall looks good! I will perform a detailed review once out of draft stage

hantangwangd · 2024-05-13T16:19:10Z

Hi @kiersten-stokes, can you rebase your branch onto the newest master branch? it's a bit too far behind :-).

steveburnett · 2024-05-14T13:54:58Z

nit, suggest adding PR # into each item of the release note entry:

== RELEASE NOTES ==

General Changes
* Upgrades fasterxml.jackson artifacts to 2.11 :pr:`22417`

Iceberg Connector Changes
* Add support for Iceberg REST catalog :pr:`22417`

presto-docs/src/main/sphinx/connector/iceberg.rst

steveburnett

LGTM! (docs)

Pull updated branch, new local docs build, looks good!

hantangwangd

Thank you for adding this great feature. I took a look at the first commit, overall it looks good to me. But I'm a little confused by the large amount of error logs printed by the test cases.

After check the code, I found that the error log is printed by RESTCatalogServlet, and by comparing the test implementations of Iceberg and Trino, I notice that they both use certain methods to hide these error messages (that is to say, the error messages are still logged, but not displayed). Iceberg inject a NOPLogger, while Trino overwrite the servlet RestCatalogServlet and injected its own configurable io.airlift.log.Logger.

So should we as well do something to get rid of these annoying error messages? What's your opinion?

ZacBlanco

mostly just nits

...iceberg/src/main/java/com/facebook/presto/iceberg/nessie/IcebergNessieSessionProperties.java

ZacBlanco · 2024-05-20T18:43:33Z

presto-iceberg/src/test/java/com/facebook/presto/iceberg/rest/IcebergRestTestUtil.java

+        restJdbcBackend.setConf(hdfsEnvironment.getConfiguration(new HdfsContext(SESSION), new Path(location.getAbsolutePath())));
+
+        Map<String, String> properties = ImmutableMap.<String, String>builder()
+                .put(URI, "jdbc:h2:mem:test" + System.nanoTime() + "_" + ThreadLocalRandom.current().nextInt())


nit: will we actually need nanoTime and a random variable? - also, maybe add another underscore between :test and `System.nanoTime()

Agreed that both is probably overkill, although this pattern came up a few different times in other tests that use this type of catalog. I was guessing it was a 'better safe than sorry' situation. Definitely agree on the underscore!

presto-iceberg/src/test/java/com/facebook/presto/iceberg/rest/TestIcebergRestConfig.java

presto-iceberg/src/test/java/com/facebook/presto/iceberg/rest/TestIcebergSmokeRest.java

presto-jdbc/src/test/java/com/facebook/presto/jdbc/TestPrestoDriverAuth.java

presto-iceberg/src/test/java/com/facebook/presto/iceberg/rest/TestIcebergDistributedRest.java

hantangwangd

I just took a whole look at all the commits, it's a very pretty refactoring, overall looks good to me. Just one thing for discussion, and some nits.

presto-iceberg/src/test/java/org/apache/iceberg/rest/IcebergRestCatalogServlet.java

presto-iceberg/src/main/java/com/facebook/presto/iceberg/rest/IcebergRestConfig.java

presto-iceberg/src/main/java/com/facebook/presto/iceberg/rest/AuthenticationType.java

presto-iceberg/src/main/java/com/facebook/presto/iceberg/rest/SessionType.java

presto-iceberg/src/test/java/org/apache/iceberg/rest/IcebergRestCatalogServlet.java

...iceberg/src/main/java/com/facebook/presto/iceberg/nessie/IcebergNessieSessionProperties.java

hantangwangd

Thanks for the fix, only some little nits. Again, it's a very pretty refactoring, make things more clearer.

presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergHiveModule.java

...to-iceberg/src/main/java/com/facebook/presto/iceberg/nessie/IcebergNessieCatalogFactory.java

presto-iceberg/src/main/java/com/facebook/presto/iceberg/rest/IcebergRestConfig.java

Make nessie config optional in session properties

hantangwangd

LGTM, thanks for the great work.

tdcmeehan

Awesome refactor and nice work--thank you!

steveburnett requested changes Apr 4, 2024

View reviewed changes

presto-docs/src/main/sphinx/connector/iceberg.rst Outdated Show resolved Hide resolved

presto-docs/src/main/sphinx/connector/iceberg.rst Outdated Show resolved Hide resolved

ZacBlanco reviewed Apr 4, 2024

View reviewed changes

steveburnett previously approved these changes Apr 4, 2024

View reviewed changes

kiersten-stokes dismissed steveburnett’s stale review via 3759ffb April 15, 2024 15:34

kiersten-stokes force-pushed the iceberg-rest branch 2 times, most recently from 3759ffb to 3667787 Compare April 15, 2024 15:41

kiersten-stokes force-pushed the iceberg-rest branch 4 times, most recently from 874c730 to abd351b Compare April 23, 2024 19:58

kiersten-stokes force-pushed the iceberg-rest branch 3 times, most recently from c6890d3 to 2a6589f Compare May 6, 2024 21:04

kiersten-stokes marked this pull request as ready for review May 8, 2024 17:55

kiersten-stokes requested review from a team and hantangwangd as code owners May 8, 2024 17:55

kiersten-stokes requested a review from presto-oss May 8, 2024 17:55

tdcmeehan self-assigned this May 13, 2024

kiersten-stokes force-pushed the iceberg-rest branch from 2a6589f to 52a4410 Compare May 14, 2024 00:30

kiersten-stokes force-pushed the iceberg-rest branch from 52a4410 to 8f7bf77 Compare May 14, 2024 14:42

kiersten-stokes mentioned this pull request May 15, 2024

Update io.jsonwebtoken artifacts for Java 11 compatibility #22762

Merged

6 tasks

kiersten-stokes requested a review from ZacBlanco May 15, 2024 19:24

kiersten-stokes commented May 15, 2024

View reviewed changes

presto-docs/src/main/sphinx/connector/iceberg.rst Outdated Show resolved Hide resolved

kiersten-stokes force-pushed the iceberg-rest branch from 8f7bf77 to 3c95a7c Compare May 15, 2024 21:58

steveburnett previously approved these changes May 16, 2024

View reviewed changes

hantangwangd reviewed May 17, 2024

View reviewed changes

ZacBlanco reviewed May 20, 2024

View reviewed changes

kiersten-stokes dismissed steveburnett’s stale review via 0f9baa2 May 20, 2024 23:21

kiersten-stokes force-pushed the iceberg-rest branch from 3c95a7c to 0f9baa2 Compare May 20, 2024 23:21

hantangwangd reviewed May 22, 2024

View reviewed changes

Add support for REST catalog in Iceberg connector

de8aa2a

kiersten-stokes force-pushed the iceberg-rest branch from 0f9baa2 to 560e4b9 Compare May 23, 2024 00:01

hantangwangd reviewed May 23, 2024

View reviewed changes

Re-factor Iceberg catalog implementation

7062bba

Make nessie config optional in session properties

kiersten-stokes force-pushed the iceberg-rest branch from 560e4b9 to 7062bba Compare May 23, 2024 22:44

hantangwangd approved these changes May 24, 2024

View reviewed changes

tdcmeehan approved these changes May 24, 2024

View reviewed changes

tdcmeehan merged commit ba8ce85 into prestodb:master May 24, 2024
57 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for REST catalog in Iceberg connector #22417

Add support for REST catalog in Iceberg connector #22417

kiersten-stokes commented Apr 3, 2024 •

edited

steveburnett left a comment

ZacBlanco left a comment •

edited

steveburnett left a comment

hantangwangd commented Apr 6, 2024

kiersten-stokes commented Apr 8, 2024

ZacBlanco commented Apr 16, 2024 •

edited

hantangwangd commented May 13, 2024

steveburnett commented May 14, 2024

steveburnett left a comment

hantangwangd left a comment

ZacBlanco left a comment

ZacBlanco May 20, 2024

kiersten-stokes May 20, 2024

hantangwangd left a comment

hantangwangd left a comment

hantangwangd left a comment

tdcmeehan left a comment

Add support for REST catalog in Iceberg connector #22417

Add support for REST catalog in Iceberg connector #22417

Conversation

kiersten-stokes commented Apr 3, 2024 • edited

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

steveburnett left a comment

Choose a reason for hiding this comment

ZacBlanco left a comment • edited

Choose a reason for hiding this comment

steveburnett left a comment

Choose a reason for hiding this comment

hantangwangd commented Apr 6, 2024

kiersten-stokes commented Apr 8, 2024

ZacBlanco commented Apr 16, 2024 • edited

hantangwangd commented May 13, 2024

steveburnett commented May 14, 2024

steveburnett left a comment

Choose a reason for hiding this comment

hantangwangd left a comment

Choose a reason for hiding this comment

ZacBlanco left a comment

Choose a reason for hiding this comment

ZacBlanco May 20, 2024

Choose a reason for hiding this comment

kiersten-stokes May 20, 2024

Choose a reason for hiding this comment

hantangwangd left a comment

Choose a reason for hiding this comment

hantangwangd left a comment

Choose a reason for hiding this comment

hantangwangd left a comment

Choose a reason for hiding this comment

tdcmeehan left a comment

Choose a reason for hiding this comment

kiersten-stokes commented Apr 3, 2024 •

edited

ZacBlanco left a comment •

edited

ZacBlanco commented Apr 16, 2024 •

edited