Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

version control memory consumption imrovement #1051

Open
pofider opened this issue Apr 17, 2023 · 0 comments
Open

version control memory consumption imrovement #1051

pofider opened this issue Apr 17, 2023 · 0 comments

Comments

@pofider
Copy link
Contributor

pofider commented Apr 17, 2023

Big workspace can cause big diffs and OOM. The problem is in total size, not in single entity diff.

The current diff blob is JSON.stringified array of all commit changes with blobs. This can be too much memory to read and parse at once.
We should create chunks based on some max reasonable size. The version entity would then have prop with a list of chunks' blob names. Additionally to the chunks, there should be one extra blob with information about which entity can be found in which chunk.

We should never need to keep all the diffs in memory. When sending to the client the local changes, we should send only entity paths with action, not diffs. This should be loaded only when the user clicks.

Applying patches should iterate chunks one by one and apply changes so we don't keep all patches in memory, only states.

This is a lot of work, I am not sure if this is worth the effort because it only affects in some way >200MB workspaces.
Perhaps we should instead invest in git integration and explain the size limits for our lightweight version control backend implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant