-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
reduce latency of list operations in snapshot browser #228
Comments
Hey, I'm curious how long The alternative approach that Backrest could possibly take would be a long pause on opening the file browser to index all paths (e.g. in a trie in the database) before permitting browsing but slow operations with more "immediate" startup is the tradeoff I've made for now. If you need to do a lot of operations, restic supports
Mind opening another bug scoped to this issue? It helps to keep discussion somewhat scoped -- bulleted bugs are difficult to address wholesale. Is the main issue that the snapshot browser closes itself after you select a file to restore? |
Hmm, 6735 items is actually quite a small file count (I'm guessing a small number of large files). I'll benchmark how long taking a listing of a full repo with a few hundred thousand files takes this weekend to get a sense of how much time indexing the whole repo takes. If it's not too bad, indexing upfront may be a reasonable way to go here. Just to check -- it looks like you're using a repo on a local HDD? I wouldn't expect this to take 20 seconds, it seems like something strange is going on there. It typically takes ~10 seconds for me using a remote target (backblaze storage) in a 1TB repo. Reading https://github.com/restic/restic/blob/228b35f074ddf4dec6ce1aea51ccfc2c413d0a01/cmd/restic/cmd_ls.go#L259-L411 I think restic is doing an optimized traversal of the file tree in the data structure to avoid reading subdirectories until they're requested so there's definitely some tradeoff to be had here in terms of how much we prefetch. |
Ran some benchmarks on a repo in backblaze (B2) so network fetch time is included in this test
following up with
I reset the cache before the first and second operations by running
tl;dr hard to say what the right tradeoff is here but I'm thinking snapshot browsing is in a pretty acceptable range. There may be something going on on your system that's degrading snapshot listing speed. Are you using a local storage repo / is your disk under heavy load? Any other factors that might contribute to the slowdown? |
I'm using a an externally HDD connected through USB. This disk is only used for my backups, so most accesses are through backrest. |
Hey, it's also taking on the order of 10 seconds with B2 as a remote on my interface or on the order of 2-5 seconds when using a local repo on an SSD. Is the device you're running Backrest on memory constrained? I wonder if the fork'd restic processes are starting slowly or hitting memory pressure as starting restic for each list operation can be an expensive. At a high level, I think I'll probably aim to keep list the way it is now as the implementation is very simple and I think works well enough on most devices (<10 seconds is acceptable latency IMO as restores are uncommon), but we can look into debugging why we're seeing such slow listings on your installation. |
I would like to propose a few improvements for the file restore:
When opening this browse and restore area, it can take quite a while for the information to show in the backrest interface. Tried the following:
It does not seem long, but the problem is if we want to restore a file or folder that is deep inside the folder structure, these times will add up. For example, to restore a file/folder that is 5 folders deep, we need to open 6 folded items until we see the file/folder, each taking around 20s (for the later example). The more complicated folder structure the more time we wait for the Backrest interface to show the desired items/folders to restore. It would be nice if we could get a faster structure update, maybe caching the contents, even if we have to wait a bit longer the 1st time we open the backup and restore view to get the full list of items, so every new folder opening would be just loading the already collected information on the interface.
The text was updated successfully, but these errors were encountered: