New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[#3259] feat(spark-connector): Support atomic create or replace Iceberg table #3260
base: main
Are you sure you want to change the base?
Conversation
3388498
to
388df1f
Compare
…lace Iceberg table
388df1f
to
6bde9ff
Compare
Hi @FANNG1 could you help review this PR? Thank you. |
ok |
@@ -416,4 +489,11 @@ private SparkMetadataColumnInfo[] getIcebergMetadataColumns() { | |||
new SparkMetadataColumnInfo("_deleted", DataTypes.BooleanType, false) | |||
}; | |||
} | |||
|
|||
private List<SparkTableInfo.SparkColumnInfo> getSimpleTableColumnWithoutComments() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please add comment why add this method and when to use
String tableName = "test_atomic_create_table_as_select"; | ||
dropTableIfExists(tableName); | ||
|
||
CatalogPlugin sparkIcebergCatalog = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is it necessary to add this check?
@@ -96,6 +103,28 @@ public UnboundFunction loadFunction(Identifier ident) throws NoSuchFunctionExcep | |||
return ((SparkCatalog) sparkCatalog).loadFunction(ident); | |||
} | |||
|
|||
@SuppressWarnings("deprecation") | |||
@Override | |||
public StagedTable stageCreate( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gravitino doesn't support stage create for now, this will bypass the gravitino when create&replace table as select
, I prefer to delay support of stage create for now. WDYT? cc @jerryshao
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gravitino doesn't support stage create for now, this will bypass the gravitino when
create&replace table as select
, I prefer to delay support of stage create for now. WDYT? cc @jerryshao
@FANNG1 But rest apis don't support RTAS. Is the purpose an audit?
What changes were proposed in this pull request?
Support atomic create or replace Iceberg table.
Why are the changes needed?
Support atomic create or replace Iceberg table.
Fix: #3259
Does this PR introduce any user-facing change?
No.
How was this patch tested?
New ITs.