You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Big workspace can cause big diffs and OOM. The problem is in total size, not in single entity diff.
The current diff blob is JSON.stringified array of all commit changes with blobs. This can be too much memory to read and parse at once.
We should create chunks based on some max reasonable size. The version entity would then have prop with a list of chunks' blob names. Additionally to the chunks, there should be one extra blob with information about which entity can be found in which chunk.
We should never need to keep all the diffs in memory. When sending to the client the local changes, we should send only entity paths with action, not diffs. This should be loaded only when the user clicks.
Applying patches should iterate chunks one by one and apply changes so we don't keep all patches in memory, only states.
This is a lot of work, I am not sure if this is worth the effort because it only affects in some way >200MB workspaces.
Perhaps we should instead invest in git integration and explain the size limits for our lightweight version control backend implementation.
The text was updated successfully, but these errors were encountered:
Big workspace can cause big diffs and OOM. The problem is in total size, not in single entity diff.
The current diff blob is JSON.stringified array of all commit changes with blobs. This can be too much memory to read and parse at once.
We should create chunks based on some max reasonable size. The
version
entity would then have prop with a list of chunks' blob names. Additionally to the chunks, there should be one extra blob with information about which entity can be found in which chunk.We should never need to keep all the diffs in memory. When sending to the client the local changes, we should send only entity paths with action, not diffs. This should be loaded only when the user clicks.
Applying patches should iterate chunks one by one and apply changes so we don't keep all patches in memory, only states.
This is a lot of work, I am not sure if this is worth the effort because it only affects in some way >200MB workspaces.
Perhaps we should instead invest in git integration and explain the size limits for our lightweight version control backend implementation.
The text was updated successfully, but these errors were encountered: