Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka executor can't leave "key" field unset #1175

Open
ageric opened this issue Aug 18, 2022 · 0 comments
Open

Kafka executor can't leave "key" field unset #1175

ageric opened this issue Aug 18, 2022 · 0 comments

Comments

@ageric
Copy link

ageric commented Aug 18, 2022

Describe the bug
It's impossible to set empty for the kafka "key" field. This is annoying, since key is supposed to be a bytefield and can happily be left unset to avoid specific broker allocation.

To Reproduce
Configure a job to run on the kafka executor with the following executor config as follows:
'executor_config': {'brokerAddress': 'brokerURL:9092', 'key': '', 'message': '{}', 'topic': 'queue'},
Then try each of the following:

  • Leave key out of the configuration.
  • Set the key to null (in the json field).
  • Set the key to the empty string.

The empty string at least gets past the point of API ingeestion and I guess it sort of works sort of as intended, although using an empty string for this field makes absolutely no sense in any way shape or form so setting it as such could be used as a workaround to at least leave it unset.

Expected behavior
Using the example python kafka consumer from https://kafka-python.readthedocs.io/en/master/usage.html (modified for broker url and message) and kafka's own topic producer like so:

echo {} | /opt/kafka/bin/kafka-console-producer.sh --topic aer01_findity.job.cronEvent --bootstrap-server

produces the following message in the consumer:

queue:1:0: key=None value=b'{}'

I.e., key remains unset.

Screenshots
Using the same kafka consumer but consuming a message from the dkron kafka executor

queue:5:5: key=b'' value=b'{}'

I.e., key is set to the null string instead of being left as NULL

** Specifications:**

  • OS: Linux
  • Version: Ubuntu 20.04, kafka 2.13, dkron 3.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant