Compressed size is not used for validating message in producer when compression is enabled #2142
Comments
|
Any updates on this issue? Thanks |
|
@parikdepa the issue is correct, unfortunately Sarama does a local compare of the pre-compression bytes, and it's probably non-trivial to fix at the moment. However, this was always (as far as I know) a convenient client-side safety net, and the remote kafka cluster will reject the produce request if you send a compressed message that it is larger than the configured maximum size that it can accept. You could just set |
This comment was marked as outdated.
This comment was marked as outdated.
github-actions
bot
added
the
stale
This was referenced May 8, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Versions
Sarama Version : v1.29.1
Go version: 1.13
Configuration
What configuration values are you using for Sarama and Kafka?
Logs
kafka: Failed to produce message to topic: kafka server: Message was too large, server rejected it to avoid allocation error.Problem Description
Even after setting producer compression by default to
sarama.CompressionGZIP. We are noticing in asyn_producer.go message byte size validation checks against uncompressed message even though Kafka broker allows compressed message based on max size settings . This is cause issues sending message that has very low compressed size compared to uncompressed message. Can we get some help on it?https://github.com/Shopify/sarama/blob/6693712f54b76066ea239255c30585832983947d/async_producer.go#L367
The text was updated successfully, but these errors were encountered: