Support procedure expire_snapshots for iceberg #22609

hantangwangd · 2024-04-25T04:11:38Z

Description

This PR support the procedure expire_snapshots for iceberg. It can be used to remove older snapshots or specified snapshots and their files which are no longer needed.

See examples as follow:

Remove snapshots older than specific day and time, but retain the last 10 snapshots::

CALL iceberg.system.expire_snapshots('schema_name', 'table_name', TIMESTAMP '2023-08-31 00:00:00.000', 10);

Remove snapshots with snapshot ID 10001 and 10002 (note that this snapshot ID should not be the current snapshot)::

CALL iceberg.system.expire_snapshots(schema => 'schema_name', table_name => 'table_name', snapshot_ids => ARRAY[10001, 10002]);

Motivation and Context

Support expiring snapshots for iceberg

Test Plan

Newly added test cases in TestExpireSnapshotProcedure

Contributor checklist

Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

Release Notes

== RELEASE NOTES ==


Iceberg Connector Changes
* Add procedure `expire_snapshots` to remove old snapshots in Iceberg. :pr:`22609`

github-actions · 2024-04-25T04:12:35Z

Codenotify: Notifying subscribers in CODENOTIFY files for diff 152f962...5fbfac8.

No notifications.

steveburnett

Thanks for the doc! Looks good, I made a few suggestions. Let me know what you think.

presto-docs/src/main/sphinx/connector/iceberg.rst

steveburnett · 2024-04-25T14:40:39Z

presto-docs/src/main/sphinx/connector/iceberg.rst

+``retain_last``                  int             Number of ancestor snapshots to preserve regardless of older_than
+                                                 (defaults to 1)
+
+``snapshot_ids``                 array of long   Array of snapshot IDs to expire


Is a word missing after "array of long"?

No, what I want to express is that the type is array of long. Is it OK with this expression in your opinion?

That's fine! I was unfamiliar with the phrase and I should have looked it up first. This is good as is. Thanks!

presto-docs/src/main/sphinx/connector/iceberg.rst

steveburnett

LGTM! (docs)

Pull updated branch, new local docs build, everything looks good. Thanks!

steveburnett · 2024-04-26T14:23:25Z

Suggest minor changes to the release note entry, following the Order of Changes and Section naming in Release Notes Guidelines.

== RELEASE NOTES ==

Iceberg Connector Changes
* Add procedure `expire_snapshots` to remove old snapshots in Iceberg. :pr:`22609`

hantangwangd · 2024-04-26T16:10:59Z

@steveburnett Thanks for the standardization of the release note, fixed!

kiersten-stokes

LGTM! All the expiry options seem at first like they're going to require a lot of explanation, but it's very clear and concise 😄

tdcmeehan

Nice work! Just a couple of questions.

...iceberg/src/test/java/com/facebook/presto/iceberg/procedure/TestExpireSnapshotProcedure.java

tdcmeehan · 2024-05-01T17:31:54Z

...iceberg/src/test/java/com/facebook/presto/iceberg/procedure/TestExpireSnapshotProcedure.java

+
+    protected long waitUntilAfter(long timestampMillis)
+    {
+        long current = System.currentTimeMillis();


This is run in a hot loop. I don't see where we actually wait though--this seems to always wait until after the timestamp of the snapshot. Do we need it?

I think we need this. Iceberg's ExpireSnapshots.expireOlderThan(timestamp) will expires all snapshots older than the given milliseconds timestamp, that means the snapshot with the same timestamp would not be expired. So for the sake of stability in testing, we'd better wait for this millisecond to pass, in case that all things happen in the same millisecond.

Can we not just use Thread.sleep(1) in place of this?

Oh, I referred to Iceberg's test cases which use this spinning wait. In my opinion, when the wait time is very short, for example shorter than 1ms in this case, a spinning wait may be better than a block wait. It could avoid thread scheduling and context switching which may be triggered by Thread.sleep(ts), that might cause the waiting time more longer than expected.

Besides, should we obey the test standards to avoid using Thread.sleep? Referring to https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md?plain=1#L374

If it's shorter than 1ms, let's add a precondition which asserts that, so that the test fails rather than busy waits? Perhaps a boundary of ~10ms is reasonable?

Sure, that makes sense! Fixed.

ZacBlanco

a few minor things, otherwise LGTM

presto-docs/src/main/sphinx/connector/iceberg.rst

...iceberg/src/test/java/com/facebook/presto/iceberg/procedure/TestExpireSnapshotProcedure.java

ZacBlanco · 2024-05-02T12:22:57Z

...iceberg/src/test/java/com/facebook/presto/iceberg/procedure/TestExpireSnapshotProcedure.java

+
+    protected long waitUntilAfter(long timestampMillis)
+    {
+        long current = System.currentTimeMillis();


Can we not just use Thread.sleep(1) in place of this?

tdcmeehan

Just one nit, otherwise LGTM

...iceberg/src/test/java/com/facebook/presto/iceberg/procedure/TestExpireSnapshotProcedure.java

hantangwangd requested review from steveburnett and a team as code owners April 25, 2024 04:11

hantangwangd requested a review from presto-oss April 25, 2024 04:11

hantangwangd requested review from ZacBlanco, kiersten-stokes and tdcmeehan April 25, 2024 07:37

steveburnett requested changes Apr 25, 2024

View reviewed changes

tdcmeehan self-assigned this Apr 25, 2024

steveburnett mentioned this pull request Apr 25, 2024

[docs] Use a separate chapter or subchapter for describing Iceberg procedures #22608

Open

hantangwangd force-pushed the support_expire_snapshots branch from 018a29c to efb9853 Compare April 26, 2024 01:14

steveburnett previously approved these changes Apr 26, 2024

View reviewed changes

kiersten-stokes previously approved these changes Apr 26, 2024

View reviewed changes

tdcmeehan reviewed May 1, 2024

View reviewed changes

hantangwangd dismissed stale reviews from kiersten-stokes and steveburnett via 7df0a18 May 2, 2024 02:45

hantangwangd force-pushed the support_expire_snapshots branch from efb9853 to 7df0a18 Compare May 2, 2024 02:45

ZacBlanco reviewed May 2, 2024

View reviewed changes

hantangwangd force-pushed the support_expire_snapshots branch 2 times, most recently from 938433a to c55b898 Compare May 7, 2024 00:31

ZacBlanco previously approved these changes May 7, 2024

View reviewed changes

tdcmeehan reviewed May 7, 2024

View reviewed changes

...iceberg/src/test/java/com/facebook/presto/iceberg/procedure/TestExpireSnapshotProcedure.java Outdated Show resolved Hide resolved

hantangwangd added 2 commits May 8, 2024 01:08

Move iceberg procedures and tests to package ...iceberg.procedure

c6909d1

Support procedure expire_snapshots for iceberg

5fbfac8

hantangwangd dismissed ZacBlanco’s stale review via 5fbfac8 May 7, 2024 17:28

hantangwangd force-pushed the support_expire_snapshots branch from c55b898 to 5fbfac8 Compare May 7, 2024 17:28

tdcmeehan approved these changes May 7, 2024

View reviewed changes

hantangwangd merged commit f78f6f1 into prestodb:master May 7, 2024
57 checks passed

hantangwangd deleted the support_expire_snapshots branch May 7, 2024 23:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support procedure expire_snapshots for iceberg #22609

Support procedure expire_snapshots for iceberg #22609

hantangwangd commented Apr 25, 2024 •

edited

github-actions bot commented Apr 25, 2024 •

edited

steveburnett left a comment

steveburnett Apr 25, 2024

hantangwangd Apr 26, 2024

steveburnett Apr 26, 2024

steveburnett left a comment

steveburnett commented Apr 26, 2024

hantangwangd commented Apr 26, 2024

kiersten-stokes left a comment

tdcmeehan left a comment

tdcmeehan May 1, 2024

hantangwangd May 2, 2024

ZacBlanco May 2, 2024

hantangwangd May 2, 2024

tdcmeehan May 6, 2024 •

edited

hantangwangd May 7, 2024

ZacBlanco left a comment

ZacBlanco May 2, 2024

tdcmeehan left a comment

Support procedure expire_snapshots for iceberg #22609

Support procedure expire_snapshots for iceberg #22609

Conversation

hantangwangd commented Apr 25, 2024 • edited

Description

Motivation and Context

Test Plan

Contributor checklist

Release Notes

github-actions bot commented Apr 25, 2024 • edited

steveburnett left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveburnett left a comment

Choose a reason for hiding this comment

steveburnett commented Apr 26, 2024

hantangwangd commented Apr 26, 2024

kiersten-stokes left a comment

Choose a reason for hiding this comment

tdcmeehan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdcmeehan May 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZacBlanco left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdcmeehan left a comment

Choose a reason for hiding this comment

hantangwangd commented Apr 25, 2024 •

edited

github-actions bot commented Apr 25, 2024 •

edited

tdcmeehan May 6, 2024 •

edited