Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow to have a dedicated key for headers content #498

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ttych
Copy link

@ttych ttych commented Nov 13, 2023

Hi Maintainers !

It is a pull request, to extend a current feature.

It is possible for some inputs to inject kafka message headers into the message (merge).
But for some scenario, there can be some overlap between kafka message keys and headers keys,
and headers overwrite message content during merge with headers content.

Even the add_headers was not present in the basic kafka input.

So the purpose of this PR is to :

  • have the add_headers feature available in every inputs
  • add another variable to control the header merge into record
    • if headers_key is set, the headers content will be pushed into the record under this key

I tried to add some tests,
while tests are working fine (with my local kafka instance) with test/plugin/test_in_kafka.rb,
tests are flappy with test/plugin/test_in_kafka_group.rb (I don't know if it is related to the kafka settings on my desktop or related to the kafka input processing).

Thanks,

Regards,

Signed-off-by: Thomas Tych <thomas.tych@gmail.com>
@ttych
Copy link
Author

ttych commented Nov 29, 2023

Hi,

Don't hesitate to comment, amend, reject the PR ;)
My motivation behind this is to be able to split and identify what comes from the message "body" and what comes from the message "header".
Thanks.

@ashie ashie self-requested a review December 6, 2023 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant