Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle deletes on tombstone messages #165

Open
ftisiot opened this issue Jun 27, 2022 · 1 comment · May be fixed by #302
Open

Handle deletes on tombstone messages #165

ftisiot opened this issue Jun 27, 2022 · 1 comment · May be fixed by #302

Comments

@ftisiot
Copy link

ftisiot commented Jun 27, 2022

When using the JDBC sink connector it would be useful if we could delete rows on tombstone messages.

This could be very useful when doing CDC from another database.
I implemented the case with PG-> Debezium-> Kafka-> JDBC sink -> Mysql

And the pipeline works with "soft deletes" (adding a column __is_deleted and setting it to true) but not with hard deletes. That can be achieved using the New Record State transformation and changing the transforms.unwrap.delete.handling.mode parameter

We could add it as additional parameter not to break compatibility

@davengeo
Copy link

I wonder if it is possible to reach the same capacity to delete rows as confluent jdbc driver. Basically having delete.enabled=true in combination with pk_mode=record_key is a well documented way to process tombstones.

sebinsunny added a commit that referenced this issue Jul 19, 2023
This PR adds the handle.tombstone configuration to the JDBC Sink Connector, allowing users to control how tombstone records (representing deletions in Kafka) are handled. When set to true, these null records are skipped.

#165
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants