Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[clickhouse] sink 不支持 物理删除操作,mysql实时同步clickhouse时delete报错 #209

Open
zjinwei12 opened this issue Apr 4, 2023 · 4 comments
Labels
enhancement New feature or request

Comments

@zjinwei12
Copy link

zjinwei12 commented Apr 4, 2023

mysql实时同步clickhouse时delete报错,显示 clickhouse不支持delete.
报错信息:

DirtyDataEntry[jobId='78a73601134bd9356ff37ab2282ee7f9', jobName='insert-into_default_catalog.default_database.test', operatorName='Sink: test', dirtyContent='{"rowKind":"DELETE","arity":3}', errorMessage='com.dtstack.chunjun.throwable.WriteRecordException: 
JdbcOutputFormat [insert-into_default_catalog.default_database.test] writeRecord error: when converting field[0] in Row(-D(6,7777,女))

希望直接在程序上去重一下 optimize TABLE table_name FINAL,数据库只保留最新的一条数据

@baisui1981
Copy link
Member

思路:

这个我看cc上是有这个功能的,他们应该是先去查看mysql的binlog日志把mysql的操作在clickhouse复现一遍,然后通过程序判断是否执行成功了,执行成功了判断clickhouse有数据了再执行去重语句。这个optimize可以手动开启,默认是0就不执行。改成1或者其他就开启这个功能

@baisui1981 baisui1981 changed the title mysql实时同步clickhouse时delete报错 [clickhouse] sink 不支持 物理删除操作,mysql实时同步clickhouse时delete报错 Apr 4, 2023
@baisui1981
Copy link
Member

clickhouse 删除语法

alter table test.order  delete where id=6;

@baisui1981
Copy link
Member

DTStack/chunjun#1606

@baisui1981 baisui1981 added the enhancement New feature or request label Apr 14, 2023
@baisui1981 baisui1981 added this to the v4.0.0 milestone Apr 14, 2023
@baisui1981 baisui1981 removed this from the v4.0.0 milestone Feb 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants