Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add row ID support to batch ORC reader #22615

Merged
merged 1 commit into from May 2, 2024
Merged

Add row ID support to batch ORC reader #22615

merged 1 commit into from May 2, 2024

Conversation

elharo
Copy link
Contributor

@elharo elharo commented Apr 25, 2024

Description

Previously only done in selective reader

Motivation and Context

Fill in row IDs on more paths

Impact

none

Test Plan

CI

Contributor checklist

  • Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.

Release Notes

== NO RELEASE NOTE ==

@elharo elharo force-pushed the fix2 branch 2 times, most recently from 18e37fa to 623481c Compare April 25, 2024 23:50
@elharo elharo changed the title Check preconditions on row IDs higher up stack Add row ID support to batch ORC reader Apr 25, 2024
@elharo elharo force-pushed the fix2 branch 4 times, most recently from ee5d947 to 52918b2 Compare April 29, 2024 12:36
@elharo elharo marked this pull request as ready for review April 29, 2024 15:34
@elharo elharo requested review from sdruzkin, a team and hantangwangd as code owners April 29, 2024 15:34
@elharo elharo force-pushed the fix2 branch 2 times, most recently from 34c6fdd to e11fa04 Compare April 29, 2024 17:40
sdruzkin
sdruzkin previously approved these changes Apr 29, 2024
Copy link
Collaborator

@sdruzkin sdruzkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Might want to add more tests.

@NikhilCollooru
Copy link
Contributor

Can we add some unit tests ?

NikhilCollooru
NikhilCollooru previously approved these changes May 1, 2024
@NikhilCollooru
Copy link
Contributor

There are 5 checkstyle errors. please check.

[INFO] There are 5 errors reported by Checkstyle 8.16 with src/checkstyle/presto-checks.xml ruleset.
Error:  src/main/java/com/facebook/presto/hive/HiveUtil.java:[216,127] (blocks) LeftCurly: '{' at column 127 should be on a new line.
Error:  src/main/java/com/facebook/presto/hive/orc/DwrfSelectivePageSourceFactory.java:[54,15] (imports) UnusedImports: Unused import - com.google.common.base.Preconditions.checkArgument.
Error:  src/test/java/com/facebook/presto/hive/TestHiveUtil.java:[90,60] (blocks) LeftCurly: '{' at column 60 should be on a new line.
Error:  src/test/java/com/facebook/presto/hive/TestHiveUtil.java:[97,60] (blocks) LeftCurly: '{' at column 60 should be on a new line.
Error:  src/test/java/com/facebook/presto/hive/TestHiveUtil.java:[104,58] (blocks) LeftCurly: '{' at column 58 should be on a new line.

Copy link
Contributor Author

@elharo elharo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PTAL. Adding unit tests invalidated existing approvals.

NikhilCollooru
NikhilCollooru previously approved these changes May 2, 2024
@NikhilCollooru
Copy link
Contributor

The test added in this PR is failing . please check

Method TestHivePageSourceProvider.testCreatePageSource_withRowIDMissingPartitionComponent()[pri:0, instance:com.facebook.presto.hive.TestHivePageSourceProvider@c017175] should have thrown an exception of type class java.lang.IllegalArgumentException
	at org.testng.internal.invokers.ExpectedExceptionsHolder.noException(ExpectedExceptionsHolder.java:81)
	at org.testng.internal.invokers.TestInvoker.considerExceptions(TestInvoker.java:853)
	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:714)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:220)
	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:945)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:193)
	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

[INFO] 
[INFO] Results:
[INFO] 
Error:  Failures: 
Error:    TestHivePageSourceProvider.testCreatePageSource_withRowIDMissingPartitionComponent » Test 
Method TestHivePageSourceProvider.testCreatePageSource_withRowIDMissingPartitionComponent()[pri:0, instance:com.facebook.presto.hive.TestHivePageSourceProvider@c017175] should have thrown an exception of type class java.lang.IllegalArgumentException
[INFO] 
Error:  Tests run: 2976, Failures: 1, Errors: 0, Skipped: 90

@elharo
Copy link
Contributor Author

elharo commented May 2, 2024

Fixed. Unfortunately that invalidates existing approvals. :-(

@elharo
Copy link
Contributor Author

elharo commented May 2, 2024

Remaining failure looks flaky, rerunning

@NikhilCollooru NikhilCollooru merged commit 8081e0c into master May 2, 2024
56 checks passed
@elharo elharo deleted the fix2 branch May 2, 2024 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants