Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new type of Serializer for supporting zero allocation scenarios #2177

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

nmosafi
Copy link

@nmosafi nmosafi commented Jan 22, 2024

Right now ISerializer and ISerializerAsync are forced to allocate a new array for every message which is sent. For a high throughput application we'd like to avoid allocating large amounts of memory.

The API exposed by librdkafka supports buffer reuse as it accepts a length and offset alongside each byte array, however this is not exposed by the serialization framework in Kafka dotnet.

The simplest approach here is to allow ArraySegment<byte> to be returned by the serializer rather than byte[], however changing all serializers to return ArraySegment<byte> would surely be a large breaking change. So instead I decided to implement this as a new type of serializer.

Please give me some feedback and thoughts on this PR.

Thanks

nmosafi and others added 2 commits January 22, 2024 22:18
Right now `ISerializer` and `ISerializerAsync` are forced to allocate a new array for every message which is sent. For a high throughput application we'd like to avoid allocating large amounts of memory.

The API exposed by librdkafka supports buffer reuse as it accepts a length and offset alongside each byte array, however this is not exposed by the serialization framework in Kafka dotnet.

The simplest approach here is to allow `ArraySegment<byte>` to be returned by the serializer rather than `byte[]`, however changing all serializers to return `ArraySegment<byte>`` would surely be a large breaking change. So instead I decided to implement this as a new type of serializer.
Copy link

cla-assistant bot commented Jan 22, 2024

CLA assistant check
All committers have signed the CLA.

@nanov
Copy link

nanov commented Mar 30, 2024

Is there a reason why this is not looked at, such functionality could allow, potentially, huge performance benefits in scenarios where the implementations relays on pooled arrays ( only minimum length is guaranteed, array may be bigger depending on the pool ) or ring buffers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants