Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-35167][cdc-connector] Introduce MaxCompute pipeline DataSink #3254

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

[FLINK-35167][cdc-connector] Introduce MaxCompute pipeline DataSink #3254

wants to merge 11 commits into from

Conversation

dingxin-tech
Copy link

No description provided.

kevinwangcs
kevinwangcs reviewed Apr 25, 2024
View reviewed changes
...ute/src/main/java/org/apache/flink/cdc/connectors/maxcompute/sink/MaxComputeEventWriter.java
import java.util.concurrent.ExecutionException;

/** a {@link SinkWriter} for {@link Event} for MaxCompute. */
public class MaxComputeEventWriter implements SinkWriter<Event> {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please consider changing "Event" to a generic type to support various serializer implementations, similar to the StarRocksSink EventRecordSerializationSchema and DorisEventSerializer implementations in Flink CDC?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am very willing to provide a sink that supports generic types. However, the current code framework would require significant changes to implement this feature. This functionality may appear in the future in flink-connector-maxcompute rather than here in flink-cdc-connector.

@github-actions github-actions bot added the docs Improvements or additions to documentation label May 6, 2024
@loserwang1024
Copy link
Contributor

I am wondering how a commercial database sink like MaxCompute to do e2e test?

@dingxin-tech
Copy link
Author

I am wondering how a commercial database sink like MaxCompute to do e2e test?

I will soon be working on creating a Docker image for a MaxCompute Emulator that launches a mocked version of MaxCompute. This will allow for end-to-end testing to be performed by initializing this image prior to regression testing.

@dingxin-tech
Copy link
Author

hi @loserwang1024 , I recently completed the development and release of a Docker image for the MaxCompute Emulator, and I have also added some related regression cases. Could you help me review the code in this pull request?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kevinwangcs kevinwangcs kevinwangcs left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
composer docs Improvements or additions to documentation e2e-tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@dingxin-tech @loserwang1024 @kevinwangcs @zhangdingxin484