[Fix][Connector-V2] Field information lost during Paimon DataType and SeaTunnel Column conversion #6767

xiaochen-zhou · 2024-04-28T05:36:53Z

Purpose of this pull request

Field information lost during Paimon DataType and SeaTunnel Column conversion, such as: field name

Does this PR introduce any user-facing change?
no

How was this patch tested?
exists test

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

# Conflicts: # docs/en/connector-v2/sink/Doris.md

Hisoka-X · 2024-04-28T07:20:37Z

Any test case?

xiaochen-zhou · 2024-04-29T06:28:53Z

Any test case?

OK, thanks for your review. I'm going to add some test cases

dailai

LGTM

Hisoka-X · 2024-05-07T02:04:29Z

...aimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/utils/RowConverter.java

@@ -320,7 +321,7 @@ public static SeaTunnelRow convert(InternalRow rowData, SeaTunnelRowType seaTunn
                    break;
                default:
                    throw new PaimonConnectorException(
-                            CommonErrorCodeDeprecated.UNSUPPORTED_DATA_TYPE,
+                            CommonErrorCode.UNSUPPORTED_DATA_TYPE,


Please throw common error by use this method.

seatunnel/seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonError.java

Line 90 in a5609d6

public static SeaTunnelRuntimeException unsupportedDataType(

Please throw common error by use this method.

seatunnel/seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonError.java

Line 90 in a5609d6

public static SeaTunnelRuntimeException unsupportedDataType(

done.

Hisoka-X · 2024-05-09T03:06:55Z

seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonError.java

+    public static SeaTunnelRuntimeException unsupportedDataType(
+            String identifier, String dataType) {
+        Map<String, String> params = new HashMap<>();
+        params.put("identifier", identifier);
+        params.put("dataType", dataType);
+        return new SeaTunnelRuntimeException(UNSUPPORTED_DATA_TYPE_SIMPLE, params);
+    }


Please reuse public static SeaTunnelRuntimeException unsupportedDataType( String identifier, String dataType, String field) . Because we should make sure user know which field can not supported.

Hisoka-X · 2024-05-09T03:08:40Z

seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonError.java

+        return new SeaTunnelRuntimeException(UNSUPPORTED_ARRAY_GENERIC_TYPE, params);
+    }
+
+    public static SeaTunnelRuntimeException unsupportedRowKind(String identifier, String rowKind) {


we need tableId when invoke unsupportedRowKind method.

we need tableId when invoke unsupportedRowKind method.

OK, I will optimize the code based on your suggestions

we need tableId when invoke unsupportedRowKind method.

OK, I will optimize the code based on your suggestions

done.

Hisoka-X · 2024-05-09T03:08:53Z

seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonError.java

+    public static SeaTunnelRuntimeException unsupportedArrayGenericType(
+            String identifier, String dataType) {
+        Map<String, String> params = new HashMap<>();
+        params.put("identifier", identifier);
+        params.put("dataType", dataType);
+        return new SeaTunnelRuntimeException(UNSUPPORTED_ARRAY_GENERIC_TYPE, params);
+    }


ditto

done.

# Conflicts: # seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonError.java # seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonErrorCode.java

xiaochen-zhou · 2024-05-13T00:59:38Z

...n/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/utils/RowTypeConverter.java

@@ -445,9 +496,8 @@ public SeaTunnelDataType<?> visit(RowType rowType) {

        @Override
        protected SeaTunnelDataType defaultMethod(DataType dataType) {


@Hisoka-X This defaultMethod method inherited from paimon seems to be difficult to obtain the fieldname.

Please put fieldname as UNKNOWN.

Please put fieldname as UNKNOWN.

done.

EricJoy2048 · 2024-05-15T06:22:51Z

...n/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/utils/RowTypeConverter.java

+                    builder.length(column.getColumnLength());
+                    return builder.build();
+                case DECIMAL:
+                    int precision =


In some cases, the precision and scale obtained here may be 0, and you must handle this situation to achieve better compatibility.

In some cases, the precision and scale obtained here may be 0, and you must handle this situation to achieve better compatibility.

done.

Hisoka-X · 2024-05-21T06:28:05Z

seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonErrorCode.java

-
-    OPERATION_NOT_SUPPORTED("COMMON-26", "<identifier> <operation> is unsupported.");
+    OPERATION_NOT_SUPPORTED("COMMON-26", "<identifier> <operation> is unsupported."),
+    UNSUPPORTED_DATA_TYPE_SIMPLE("COMMON-27", "'<identifier>' unsupported data type '<dataType>'"),


Suggested change

UNSUPPORTED_DATA_TYPE_SIMPLE("COMMON-27", "'<identifier>' unsupported data type '<dataType>'"),

dailai · 2024-05-22T02:11:42Z

@xiaochen-zhou Please resolve the conflicts.

dailai · 2024-05-22T07:01:27Z

@xiaochen-zhou Please resolve the conflicts.

# Conflicts: # seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonErrorCode.java # seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/catalog/PaimonCatalog.java

xiaochen-zhou · 2024-05-22T12:35:09Z

@xiaochen-zhou Please resolve the conflicts.

done.

Hisoka-X

LGTM if ci passed.

xiaochen-zhou · 2024-05-22T23:23:56Z

LGTM if ci passed.

done.

ClownXC added 4 commits April 27, 2024 13:44

paimon catalog basetype

6c102ae

replace dead links

4a5431f

Merge branch 'dev' into paimon_catalog_basetype

fb43528

# Conflicts: # docs/en/connector-v2/sink/Doris.md

remove PaimonType

2d1cb44

Hisoka-X added bug paimon labels Apr 28, 2024

add test case

0e91978

xiaochen-zhou changed the title ~~[Fix][Connector-V2] Field information lost during Paimon and SeaTunnel row type conversion~~ [Fix][Connector-V2] Field information lost during Paimon DataType and SeaTunnel Column conversion May 1, 2024

dailai approved these changes May 7, 2024

View reviewed changes

github-actions bot added the reviewed label May 7, 2024

Hisoka-X reviewed May 7, 2024

View reviewed changes

remove CommonErrorCodeDeprecated

8036acb

Hisoka-X reviewed May 9, 2024

View reviewed changes

ClownXC added 6 commits May 12, 2024 09:21

reuse unsupportedDataType

548e68a

unsupportedArrayGenericType add fieldname

6e3f204

fix

bdb4b77

fix

ea955f3

Merge branch 'dev' into paimon_catalog_basetype

760fb29

# Conflicts: # seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonError.java # seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonErrorCode.java

optimize code

3915e36

xiaochen-zhou force-pushed the paimon_catalog_basetype branch from 2c327bf to 3915e36 Compare May 12, 2024 14:01

xiaochen-zhou commented May 13, 2024

View reviewed changes

EricJoy2048 reviewed May 15, 2024

View reviewed changes

double range validation

a7c0649

Hisoka-X reviewed May 21, 2024

View reviewed changes

remove UNSUPPORTED_DATA_TYPE_SIMPLE

e57182c

Merge branch 'dev' into paimon_catalog_basetype

b068137

# Conflicts: # seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonErrorCode.java # seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/catalog/PaimonCatalog.java

code style

f162b5e

Hisoka-X approved these changes May 22, 2024

View reviewed changes

github-actions bot added the approved label May 22, 2024

hailin0 approved these changes May 23, 2024

View reviewed changes

hailin0 merged commit 6cf6e41 into apache:dev May 23, 2024
4 checks passed

xiaochen-zhou deleted the paimon_catalog_basetype branch May 23, 2024 23:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix][Connector-V2] Field information lost during Paimon DataType and SeaTunnel Column conversion #6767

[Fix][Connector-V2] Field information lost during Paimon DataType and SeaTunnel Column conversion #6767

xiaochen-zhou commented Apr 28, 2024 •

edited

Hisoka-X commented Apr 28, 2024

xiaochen-zhou commented Apr 29, 2024

dailai left a comment

Hisoka-X May 7, 2024

xiaochen-zhou May 9, 2024

Hisoka-X May 9, 2024

Hisoka-X May 9, 2024

xiaochen-zhou May 10, 2024

xiaochen-zhou May 13, 2024

Hisoka-X May 9, 2024

xiaochen-zhou May 13, 2024

xiaochen-zhou May 13, 2024

Hisoka-X May 15, 2024

xiaochen-zhou May 20, 2024

EricJoy2048 May 15, 2024

xiaochen-zhou May 20, 2024

Hisoka-X May 21, 2024

dailai commented May 22, 2024

dailai commented May 22, 2024

xiaochen-zhou commented May 22, 2024

Hisoka-X left a comment

xiaochen-zhou commented May 22, 2024

		@@ -445,9 +496,8 @@ public SeaTunnelDataType<?> visit(RowType rowType) {

		@Override
		protected SeaTunnelDataType defaultMethod(DataType dataType) {

[Fix][Connector-V2] Field information lost during Paimon DataType and SeaTunnel Column conversion #6767

[Fix][Connector-V2] Field information lost during Paimon DataType and SeaTunnel Column conversion #6767

Conversation

xiaochen-zhou commented Apr 28, 2024 • edited

Check list

Hisoka-X commented Apr 28, 2024

xiaochen-zhou commented Apr 29, 2024

dailai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dailai commented May 22, 2024

dailai commented May 22, 2024

xiaochen-zhou commented May 22, 2024

Hisoka-X left a comment

Choose a reason for hiding this comment

xiaochen-zhou commented May 22, 2024

xiaochen-zhou commented Apr 28, 2024 •

edited