Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EXPORTER] optimize OTLP HTTP compression #2570

Open
marcalff opened this issue Feb 29, 2024 · 2 comments
Open

[EXPORTER] optimize OTLP HTTP compression #2570

marcalff opened this issue Feb 29, 2024 · 2 comments
Labels
bug Something isn't working good first issue Good for newcomers help wanted Good for taking. Extra help will be provided by maintainers

Comments

@marcalff
Copy link
Member

Investigate and/or implement possible performance improvements.

Related to this idea from #2530

Do you think we can use the in-place compression of the data without using a separate output buffer compressed_body, as suggested here - https://stackoverflow.com/questions/12398377/is-it-possible-to-have-zlib-read-from-and-write-to-the-same-memory-buffer/12412863#12412863. This is suggested by "Mark Adler", the author of zlib, and would be better memory optimzation in the hot-path of upload.

@marcalff marcalff added the bug Something isn't working label Feb 29, 2024
@github-actions github-actions bot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Feb 29, 2024
@marcalff marcalff added help wanted Good for taking. Extra help will be provided by maintainers good first issue Good for newcomers and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 29, 2024
Copy link

This issue is available for anyone to work on. Make sure to reference this issue in your pull request.
✨ Thank you for your contribution! ✨

@perhapsmaple
Copy link
Contributor

Hi, I looked at implementing in-place zlib compression suggested by @lalitb. A couple of issues that I need some help with:

  1. For the case where the data is insufficiently compressible, the function allocates a temporary buffer, copies the remaining uncompressed data into it, and then completes the compression from the temporary buffer into the remaining space in the output buffer. To avoid this scenario, the author recommends to set max to be equal to deflateBound(stream, len). If we follow the same advice, then do we resize http_requrest->body before every compression, or do we resize only when the function fails with Z_BUF_ERROR and then resize the buffer and retry compression?

  2. The deflate stream can be reused across multiple calls, but is not thread safe. From what I can see, the Session::SendRequest method seems to be used in a thread-safe way in the OtlpHttp exporter. If so, we can initialize the stream object once per session and reuse it for multiple requests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers help wanted Good for taking. Extra help will be provided by maintainers
Projects
None yet
Development

No branches or pull requests

2 participants