Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Subtask] [spark-connector] support passing catalog properties to spark connector #3292

Closed
Tracked by #1227
FANNG1 opened this issue May 7, 2024 · 2 comments · Fixed by #3270
Closed
Tracked by #1227

[Subtask] [spark-connector] support passing catalog properties to spark connector #3292

FANNG1 opened this issue May 7, 2024 · 2 comments · Fixed by #3270
Assignees
@FANNG1
Labels
0.5.1 Release v0.5.1 0.6.0 Release v0.6.0 subtask Subtasks of umbrella issue

Comments

@FANNG1
Copy link
Contributor

FANNG1 commented May 7, 2024

Describe the subtask

To fix #3181 , we should allow user to pass catalog properties to spark connector, like spark.sql.hive.metastore.jars

Parent issue

#1227
@FANNG1 FANNG1 added the subtask Subtasks of umbrella issue label May 7, 2024
@FANNG1 FANNG1 mentioned this issue May 7, 2024
Open
23 tasks
@FANNG1
Copy link
Contributor Author

FANNG1 commented May 7, 2024

Which properties should passed to the spark connector?

  1. properties with prefix spark.bypass.. Pros: keep consistent with gravitino.bypass, and trino.bypass, Cons: complex, especially for the properties shared by multi engines.
  2. all catalog properties. Pros: simple and clear. Cons: lack of control.

@jerryshao @qqqttt123 WDYT?

@FANNG1 FANNG1 mentioned this issue May 7, 2024
Merged
@FANNG1 FANNG1 self-assigned this May 10, 2024
jerryshao pushed a commit that referenced this issue May 13, 2024
@FANNG1
[#3292] fix(spark-connector): passing Gravitino catalog properties to…
013021d 
… spark connector (#3270)

### What changes were proposed in this pull request?
1. passing Gravitino catalog properties with `spark.bypass.` prefix to
spark connector
2. refactor the catalog properties transform logic.

### Why are the changes needed?

Fix: #3292

### Does this PR introduce _any_ user-facing change?
yes, add document

### How was this patch tested?
1. add UT
3. test in local envriment
github-actions bot pushed a commit that referenced this issue May 13, 2024
@FANNG1 @web-flow
[#3292] fix(spark-connector): passing Gravitino catalog properties to…
5651776 
… spark connector (#3270)

### What changes were proposed in this pull request?
1. passing Gravitino catalog properties with `spark.bypass.` prefix to
spark connector
2. refactor the catalog properties transform logic.

### Why are the changes needed?

Fix: #3292

### Does this PR introduce _any_ user-facing change?
yes, add document

### How was this patch tested?
1. add UT
3. test in local envriment
@jerryshao jerryshao mentioned this issue May 13, 2024
Merged
@jerryshao jerryshao added 0.5.1 Release v0.5.1 0.6.0 Release v0.6.0 labels May 13, 2024
@FANNG1
Copy link
Contributor Author

FANNG1 commented May 13, 2024

using spark.bypass to keep consistent

FANNG1 added a commit that referenced this issue May 13, 2024
@jerryshao @FANNG1
[#3292] fix(spark-connector): passing Gravitino catalog properties to…
7a6d045 
… spark connector (#3358)

### What changes were proposed in this pull request?
1. passing Gravitino catalog properties with `spark.bypass.` prefix to
spark connector
2. refactor the catalog properties transform logic.

### Why are the changes needed?

Fix: #3292

### Does this PR introduce _any_ user-facing change?
yes, add document

### How was this patch tested?
1. add UT
3. test in local envriment

Co-authored-by: FANNG <xiaojing@datastrato.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@FANNG1 FANNG1

Labels
0.5.1 Release v0.5.1 0.6.0 Release v0.6.0 subtask Subtasks of umbrella issue
Projects
None yet
Milestone
No milestone
2 participants
@jerryshao @FANNG1