New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AMORO-2540] Report expiring snapshots event for Iceberg and mixed format #2541
base: master
Are you sure you want to change the base?
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2541 +/- ##
============================================
+ Coverage 32.44% 32.61% +0.16%
- Complexity 4412 4433 +21
============================================
Files 590 596 +6
Lines 50055 50260 +205
Branches 6622 6640 +18
============================================
+ Hits 16242 16391 +149
- Misses 32533 32578 +45
- Partials 1280 1291 +11
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@XBaith I left some comments. PTAL.
ams/api/src/main/java/com/netease/arctic/ams/api/events/expire/ExpireEvent.java
Outdated
Show resolved
Hide resolved
ams/api/src/main/java/com/netease/arctic/ams/api/events/expire/ExpireEvent.java
Outdated
Show resolved
Hide resolved
ams/api/src/main/java/com/netease/arctic/ams/api/events/expire/ExpireResult.java
Outdated
Show resolved
Hide resolved
ams/api/src/main/java/com/netease/arctic/ams/api/events/expire/ExpireEvent.java
Outdated
Show resolved
Hide resolved
ams/api/src/main/java/com/netease/arctic/ams/api/events/IcebergReportEvent.java
Outdated
Show resolved
Hide resolved
ams/api/src/main/java/com/netease/arctic/ams/api/events/expire/ExpireOperation.java
Outdated
Show resolved
Hide resolved
...er/src/main/java/com/netease/arctic/server/optimizing/maintainer/IcebergTableMaintainer.java
Outdated
Show resolved
Hide resolved
# Conflicts: # ams/api/pom.xml # ams/server/src/main/java/com/netease/arctic/server/optimizing/maintainer/IcebergTableMaintainer.java # core/src/main/java/com/netease/arctic/utils/TableFileUtil.java # pom.xml
Hi, thanks for this wonderful feature. I wonder if you can somehow explain the motivation of this definition? |
The purpose is to report information about files that have been deleted after the expiration timestamp, such as the path to the file, so that downstream systems can monitor or utilize this information for appropriate operations. But after discussing with @zhoujinsong , I think the reported information should be as concise as possible, and also need to wait for the API to be refactored to handle it. |
Why are the changes needed?
Close #2540 .
Brief change log
How was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request
Documentation