BUG: performance issue when expanded directories are being regenerated

[ghostbuster91]

Did you check docs and existing issues?

• I have read all the docs.
• I have searched the existing issues.
• I have searched the existing discussions.

Neovim Version (nvim -v)

0.9.5

Operating System / Version

nixos

Describe the Bug

I noticed recently that in certain projects/situations my neovim becomes very slow.
Turned out, closing neotree solves the problem. I will enable debug logging and provide more info later, but wanted to make you aware of such problem.

Screenshots, Traceback

Eventually I was even able to get following stacktrace from neotree however I am not sure if there isn't anything besides this that slows down neotree.

Error executing luv callback:
...-neotree/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:210: bad argument #1 to 'fs_readdir' (uv_dir expected, got nil)
stack traceback:
	[C]: in function 'fs_readdir'
	...-neotree/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:210: in function <...-neotree/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:209>
Error executing luv callback:
...-neotree/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:210: bad argument #1 to 'fs_readdir' (uv_dir expected, got nil)
stack traceback:
	[C]: in function 'fs_readdir'
	...-neotree/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:210: in function <...-neotree/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:209>
Error executing luv callback:
...-neotree/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:210: bad argument #1 to 'fs_readdir' (uv_dir expected, got nil)
stack traceback:
	[C]: in function 'fs_readdir'
	...-neotree/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:210: in function <...-neotree/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:209>
Error executing luv callback:
...-neotree/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:210: bad argument #1 to 'fs_readdir' (uv_dir expected, got nil)
stack traceback:
	[C]: in function 'fs_readdir'
	...-neotree/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:210: in function <...-neotree/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:209>

Steps to Reproduce

1. Have a relatively big project where you can generate a lot of files (e.g. because of compilation process). At the moment I cannot provide a project to reproduce it with because it is a commercial one, but I will try to replicate it later on some opensource one.
2. Compile code
3. Expand directories with compiled code
4. Compile code once again and observe as your computer burns

Expected Behavior

I would expect neotree to handle such case gracefully.

Your Configuration

https://github.com/ghostbuster91/dot-files/blob/a84a394f249a8567c4c5bf1cbe39c31c13f233d4/modules/hm/neovim/lua/local/neotree/init.lua#L46

[ghostbuster91]

after turning on debug logging I can see that whenever I save a file LSP server recompiles some parts of the project which causes a wall of

e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:245: scan_dir_async - start 
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:245: scan_dir_async - start 
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:245: scan_dir_async - start 
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:245: scan_dir_async - start 
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:245: scan_dir_async - start 
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:245: scan_dir_async - start 
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish
e/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:268: scan_dir_async - finish

followed by

...ckages/start/p_nvim-neotree/lua/neo-tree/git/ignored.lua:143: Failed to load ignored files for 
...ckages/start/p_nvim-neotree/lua/neo-tree/git/ignored.lua:143: Failed to load ignored files for 
...ckages/start/p_nvim-neotree/lua/neo-tree/git/ignored.lua:143: Failed to load ignored files for 
...ckages/start/p_nvim-neotree/lua/neo-tree/git/ignored.lua:143: Failed to load ignored files for 
...ckages/start/p_nvim-neotree/lua/neo-tree/git/ignored.lua:143: Failed to load ignored files for 
...ckages/start/p_nvim-neotree/lua/neo-tree/git/ignored.lua:143: Failed to load ignored files for 
...ckages/start/p_nvim-neotree/lua/neo-tree/git/ignored.lua:143: Failed to load ignored files for 
...ckages/start/p_nvim-neotree/lua/neo-tree/git/ignored.lua:143: Failed to load ignored files for 

A lot of directories gets rescanned even though they might have been already collapsed.

[pynappo]

i recently got #1343 merged to main which should handle the errors more gracefully without causing those mass stacktraces. can you switch branches and try it out?

edit: i don't know if this would fix performance in any way, though.

[pysan3]

@Hexdigest123 I don't think your issue is related to this issue.
Your issue is related to UI rerendering of neo-tree, and I believe is caused only when expanding a directory for the first time.
Please make a new issue or please correct me if I'm wrong tho.

OP says there's high CPU usage when a large number of files are updated in the disk, (for example by compiling a project with cmake, a lot of files in build will be regenerated), and neo-tree tries to catch up to those updates. (right? @ghostbuster91)

This is understandable, and we might have to say this is an expected behavior with the current codebase, as neo-tree has to recreate the tree and rerender the whole buffer for each file update (debounced every 50 ms iirc).
And the performance becomes much worse if the directory is a git repo.

This is a big issue, but it is very difficult to fix at the same time. I'm doing a big rewrite and will incorporate with nvim-nio which will hopefully make the situation better, but it will take time.

[ghostbuster91]

yes, @pysan3 that is correct. I want to add that is is a regular thing for me that after working whole day on a project I have to restart vim because neotree becomes unbearably slow.

The filesystem source is very generic (rightfully) as it handles a lot of different cases like e.g. multiple git repositories (afaik) and other things (e.g. git not in the cwd root). fyi: I am thinking about writing my own filesystem source more optimized for my usual use-case - single git repository always in the cwd root (if at all).

[pysan3]

@ghostbuster91 Yes you are right. There are some object leaks that lua GC cannot collect and unfortunately stays forever.

I am doing a re-write to solve these problems and hopefully make it blazingly fast, but will take long.

In the meantime, could you test the following snippet to simply kill neo-tree states and should start things allover?

Please run this with all tree closed and the performance should get a little better.
But this cannot clear everything, so you'd need to restart neovim from time to time. I'm sorry about that.

require("neo-tree.events").clear_all_events()
require("neo-tree.sources.manager")._clear_state()
require("neo-tree.ui.renderer").clean_invalid_neotree_buffers(true)

collectgarbage("collect")

If you get an error, could you report it to #1367.

Sorry for the inconvenience.

Things that don't get erased

@ developers

• NuiSplit/Float object of each state
• Many cache variables