Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#3330] feat(spark-connector): use old create table interface to support mult spark version #3333

Closed
wants to merge 1 commit into from

Conversation

FANNG1
Copy link
Contributor

@FANNG1 FANNG1 commented May 11, 2024

What changes were proposed in this pull request?

use old create table interface to support mult spark version

Why are the changes needed?

the current interface was introduced in Spark 3.4
the old create table table interface had better Spark version compatibility.

Fix: #3330

Does this PR introduce any user-facing change?

no

How was this patch tested?

existing test.

@qqqttt123
Copy link
Contributor

Although we supported legacy interfaces, we still use new interfaces to implement the deprecated interfaces.

@FANNG1
Copy link
Contributor Author

FANNG1 commented May 11, 2024

Although we supported legacy interfaces, we still use new interfaces to implement the deprecated interfaces.

Sorry, I couldn't get your point :(

@jerryshao
Copy link
Collaborator

One possible way is that we implement this method in different Spark version's module, not in common module to avoid this issue?

@FANNG1
Copy link
Contributor Author

FANNG1 commented May 11, 2024

One possible way is that we implement this method in different Spark version's module, not in common module to avoid this issue?

yes, but this introduces redundant code which is located in different spark version's module. And there seems no bad effect to using the old interface.

@qqqttt123
Copy link
Contributor

Although we supported legacy interfaces, we still use new interfaces to implement the deprecated interfaces.

Sorry, I couldn't get your point :(

We should have all of these methods at same time as follows.

createTable(
      Identifier ident, StructType schema, Transform[] partitions, Map<String, String> properties);

  public Table createTable(
      Identifier ident, Column[] columns, Transform[] transforms, Map<String, String> properties)

@jerryshao
Copy link
Collaborator

jerryshao commented May 11, 2024

One possible way is that we implement this method in different Spark version's module, not in common module to avoid this issue?

yes, but this introduces redundant code which is located in different spark version's module. And there seems no bad effect to using the old interface.

No, the basic logic is the same and can be reused in different modules, you can define a common, version independent method.

@FANNG1
Copy link
Contributor Author

FANNG1 commented May 11, 2024

the new interface and Column type was introduced in Spark 3.4 which couldn't be reused in older versions.

@jerqi
Copy link
Contributor

jerqi commented May 11, 2024

the new interface and Column type was introduced in Spark 3.4 which couldn't be reused in older versions.

the new interface and Column type was introduced in Spark 3.4 which couldn't be reused in older versions.

Maybe you needn't to reuse, but we should have the similar interface. Maybe you can use object as the type. WDYT?

@FANNG1
Copy link
Contributor Author

FANNG1 commented May 13, 2024

let's delay this until after spark multi version design is finished. cc @jerryshao @qqqttt123

@FANNG1 FANNG1 closed this May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Improvement] use old create table interface to support mult spark version
4 participants