Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gensupport: allow passing in buffer for uploads to reduce allocations #638

Open
zimnx opened this issue Aug 25, 2020 · 1 comment
Open
Assignees
Labels
type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@zimnx
Copy link

zimnx commented Aug 25, 2020

When uploading lots of files using ObjectStorage.Insert(), each file allocates his own buffer. This results in increased CPU usage and high RSS memory.

Graph below shows how many space was allocated during upload of ~30k of 10MB files:
Selection_132

User should be able to pass his own buffer and be responsible for allocations or library shouldn't be so wasteful.

Pull request which adds user facing insert option is here: #632, I was asked to an open issue to discuss it.
cc. @tritone @codyoss

@codyoss codyoss changed the title High memory allocation footprint during uploading gensupport: allow passing in buffer for uploads to reduce allocations Aug 25, 2020
@codyoss codyoss added the type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. label Aug 25, 2020
@codyoss
Copy link
Member

codyoss commented Aug 25, 2020

@zimnx Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

No branches or pull requests

2 participants