-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature][Connector Hive] support hive savemode #6842
base: dev
Are you sure you want to change the base?
Conversation
@EricJoy2048 @dailai @ruanwenjun hi, guys. PTAL when you have time. |
@@ -66,7 +74,8 @@ public OptionRule optionRule() { | |||
|
|||
ReadonlyConfig finalReadonlyConfig = | |||
generateCurrentReadonlyConfig(readonlyConfig, catalogTable); | |||
return () -> new HiveSink(finalReadonlyConfig, catalogTable); | |||
CatalogTable finalCatalog = renameCatalogTable(finalReadonlyConfig, catalogTable); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
replace with target hive sink table name, if not replace here, will pass source table name to hive.
like fake
to hive sink. so when use this catalog, will has issue, replaced here
String describeFormattedTableQuery = "describe formatted " + tablePath.getFullName(); | ||
try (PreparedStatement ps = connection.prepareStatement(describeFormattedTableQuery)) { | ||
ResultSet rs = ps.executeQuery(); | ||
return processResult(rs, tablePath, builder, partitionKeys); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now hive table informaction is parse from the query result. That's not very elegant, but it work.....
.withValue( | ||
FIELD_DELIMITER.key(), | ||
ConfigValueFactory.fromAnyRef( | ||
parameters.get("field.delim"))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this line will has issue if the field.delim is \t
, the ConfigValueFactory.fromAnyRef
with replace it to \\t
. then the writted data will has issue.
大佬你好,创建的statement好像有点问题 |
是提交partition信息的语句吗 还是哪个语句? |
@@ -33,7 +33,7 @@ By default, we use 2PC commit to ensure `exactly-once` | |||
| name | type | required | default value | | |||
|-------------------------------|---------|----------|----------------| | |||
| table_name | string | yes | - | | |||
| metastore_uri | string | yes | - | | |||
| hive_jdbc_url | string | yes | - | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How to be compatible with older versions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it not compatible with old version, because i don't want both use hive2 jdbc and hive metastore. so i removed the metastore only use jdbc.
是的,当schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"时,save_mode_create_template 是必须的吗?当为 “” 时,报错为我描述的那个错误 |
是必须的, 这个是你表不存在时的要执行的建表语句 |
那如果我不知道source的表结构的话,是否该source-sink的conf就不能成立?这个和mysql的schema_save_mode 的配置实现的效果不一样吗? |
有些许的不一样 |
|
目前的代码还有几个问题:
|
Modify the hive documentation by referring to mysql.md |
- fix storage parameter encode issue - update add/drop partition sql
Purpose of this pull request
subtask of #5390
Does this PR introduce any user-facing change?
How was this patch tested?
Check list
New License Guide
release-note
.