Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support index mapping strategy #657

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

nizanifrach
Copy link

@nizanifrach nizanifrach commented Oct 2, 2022

Problem

Currently, there is no way to configure an index strategy to map a custom index name by topic and message payload.
To insert a document to a custom index name at the moment, the developer needs to create a specific topic name that corresponds with the Elasticsearch index name

#595
#385
#341

Solution

In this solution, we introduce a new configuration index.mapper which gets a classpath as an argument of a custom implementation of the IndexMapper.java interface. the getIndex() function in the interface return the custom index name.

For backward compatibility reasons, the index name creation logic was transferred to a DefaultIndexMapper.java which will be loaded if no other configuration is specified.

Does this solution apply anywhere else?
  • yes
  • no
If yes, where?

I took inspiration from the MongoDB source connector
https://github.com/mongodb/mongo-kafka/tree/master/src/main/java/com/mongodb/kafka/connect/source

Test Strategy

Unit tests:

  • JsonNode converter tests.
  • IndexMapper configuration test.
  • IndexMapper logic tests.

Integration tests:

  • All integration tests pass, keeping backward compatibility.

tested manually as well.

Testing done:
  • Unit tests
  • Integration tests
  • System tests
  • Manual tests

Release Plan

This feature is backwards compatible.
Merging to master

Unit tests:
* JsonNode converter tests
* IndexMapper configuration test
* IndexMapper logic tests

Integration tests:
* All integration tests pass, keeping backwards compatibility
@nizanifrach nizanifrach requested a review from a team as a code owner October 2, 2022 12:11
@CLAassistant
Copy link

CLAassistant commented Oct 2, 2022

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@nizanifrach nizanifrach changed the base branch from master to 11.1.x October 2, 2022 13:33
@nizanifrach nizanifrach changed the base branch from 11.1.x to master October 2, 2022 13:38
@nizanifrach nizanifrach changed the base branch from master to 11.1.x October 2, 2022 14:02
@nizanifrach nizanifrach changed the base branch from 11.1.x to 14.0.x October 2, 2022 14:37
@nizanifrach nizanifrach changed the base branch from 14.0.x to master October 2, 2022 14:39
@nizanifrach
Copy link
Author

Hi @snehashisp @arihant-confluent @aniketshrimal , Your review and input on this feature would be highly appreciated.
Thank you 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants