Ivy projectile find file / completion INCREDIBLY slow, CPU to 100%, and machine occasionally locks?

[saumets]

Observed behavior

In large project folders (400+ files?) Projectile find file w/ Ivy is INCREDIBLY slow, with CPU processing jumping to almost 100% and occasionally even locking my machine. Is this Ivy completion issue w/ projectile?

Below is a screenshot showing the behaviour. In the below screenshot I have already completely typed out src/services/submit-order-service/index.js. We can see that Ivy Projectile find file is stuck at src/services and that my CPU processing has jumped to 99.8% percent and clocking. My system in the below screenshot almost fully locked!

Ivy has become completely unusable for me. I've switched to using HELM and have no issues with helm-projectile and finding files / completion. Everything is smooth and fast with HELM. However, I'd still prefer to use Ivy 😢

Is it maybe related to this merge commit? #740

Expected behavior

Ivy projectile find file / completion (SPC p p + typing file we're searching for) to be smooth and fast. Ivy completion to be just as fast as finding regular files (SPC f .) and HELM projectile find file.

Steps to reproduce

1. Use Ivy instead of HELM, ido
2. Get a large project indexed with Projectile
3. Select the projectile project using SPC - p - p + select project
4. Begin Ivy completion to find a file deep in project folders searching for a file in a folder w/ numerous files (300-400+ files)
5. Observe CPU processing skyrocketing to 100% and machine locking 😢

System information

Am using the develop branch of DOOM and working. Currently have HELM installed and have disabled Ivy.

Please let me know if you'd like me to re-enable Ivy and do a complete debug dump!

Click to expand

Replace this line with the output of *one* of these commands:

+ `M-x doom/info` (from inside Emacs)
+ `DEBUG=1 make doctor` (command line)

[hlissner]

If it's working fine for helm, I suspect this may be Doom's fault. Try this and see if it helps anything:

(after! counsel-projectile
  (ivy-set-display-transformer
   'counsel-projectile-find-file
   'counsel-projectile-find-file-transformer))

[hlissner]

Actually, the more I think about it, the less likely Doom is to blame. The above snippet undoes the only modification Doom imposes on ivy+projectile that could interfere with the file-list building process, and counsel's default consumes the same number of cycles.

Also, 300-400 files is in fact, quite small. Doom has ~390 *.el files in modules/ alone, and it indexes almost instantly, regardless of the backend projectile uses (git-grep or find).

So let's try this:

1. Switch to the problem project manually (i.e. switch to a file inside it with M-x find-file)
2. Turn on the profiler with M-x doom/toggle-profiler
3. Invoke counsel-projectile-find-file with SPC SPC (if you've rebound it, it's also bound to SPC f /)
4. if it's taking too long, mash C-g until it aborts, otherwise try to wait for it.
5. Run M-x doom/toggle-profiler to get Doom to produce a report on what it's been doing this entire time.

That will help us figure out where Emacs is struggling!

[ocharles]

I have experienced something similar. We have a lot of files in our project and anything more than about 6 characters with the default fuzzy search grinds to a halt.

…

On Wed, 1 Aug 2018, 3:28 pm Henrik Lissner, ***@***.***> wrote: Actually, the more I think about it, the less likely Doom is to blame. The above snippet undoes the *only* modification Doom imposes on ivy+projectile that could interfere with the file-list building process, and counsel's default consumes the same number of cycles. Also, 300-400 files is in fact, quite small. Doom has ~390 files in modules/ alone, and it indexes almost instantly, regardless of the backend projectile uses (git-grep or find). So let's try this: 1. Switch to the problem project manually (i.e. switch to a file inside it with M-x find-file) 2. Turn on the profiler with M-x doom/toggle-profiler 3. Invoke counsel-projectile-find-file with SPC SPC (if you've rebound it, it's also bound to SPC f /) 4. Once it's done, or has taken too long, mash C-g until it aborts 5. Run M-x doom/toggle-profiler to get Doom to produce a report on what it's been doing this entire time. That will help us figure out where Emacs is struggling! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#774 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABRjpogQ8anoNyRgmFuephVn2SwBqpUks5uMbslgaJpZM4Vqh3p> .

[hlissner]

@ocharles Are you a windows user? Projectile falls back to an elisp implementation there, which is significantly slower.

Although that does bring up a new possibility: do the two of you have +fuzzy enabled for the :completion ivy module?

[saumets]

I am using the default init.example.el that gets updated whenever it gets updated.
This by default has (ivy +fuzzy) enabled, correct?

Of course since this issue arose I'm currently using (helm +fuzzy) 😄

Will try your profiler recommendations later this evening when I have a chance!

[hlissner]

Ah I see, yes, +fuzzy is enabled by default and is likely the cause. I don't use it due to the performance implications.

Could you try disabling it (then run bin/doom refresh)?

Fuzzy search is powered by flx, which may be the cause. I'm planning to switch to presclient, which claims to be faster, but I'd like to be sure before I do.

[saumets]

@hlissner Hmm? I'm using (helm +fuzzy) as a backup now without any issues and it is super fast and smooth.

+fuzzy would have different performance implications when used with Ivy vs HELM?

[ocharles]

No, Linux and I am using +fuzzy (and +childframe)

…

On Wed, 1 Aug 2018, 4:02 pm Henrik Lissner, ***@***.***> wrote: @ocharles <https://github.com/ocharles> Are you a windows user? Projectile falls back to an elisp implementation there, which is significantly slower. Although that does bring up a new possibility: do the two of you have +fuzzy enabled for the :completion ivy module? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#774 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABRjkrD8n-wiLbKJANhWZLQo4srhMWQks5uMcLzgaJpZM4Vqh3p> .

[hlissner]

+fuzzy would have different performance implications when used with Ivy vs HELM?

@skalfyfan It's possible. It may not be flx, even, but how ivy/helm implements it. It'll help to eliminate possibilities. If disabling +fuzzy doesn't change anything, at least it'll tell me where not to look.

[saumets]

@hlissner You got it! 💯

I just tested Ivy without +fuzzy and it works great and is super fast!! If I re-enable

(ivy
  +fuzzy)

Then it craps out again as I described. I ran make on each config modification. Seems the issue is definitely with the Ivy implementaiton of flx since Helm is fine! 🤔

[hlissner]

Alright, I've replaced flx with prescient, which I hope will be faster for +fuzzy users (not to mention, brings frecency sorting to a variety of commands).

Give it a try and let me know if it has improved the performance of counsel-projectile-find-file in large projects.

[hliebert]

I've been using counsel-fzf for fuzzy search, which is a part of ivy and using fzf. Is there any advantage to enabling +fuzzy?

[saumets]

@hlissner just pulled the latest from develop. Ran make and it deleted flx and installed ivy-prescient.

Sorry to report that performance is just as bad w/ ivy-prescient as it was withflx, maybe even slightly worse! 😭 I still have Activity Monitor open while typing this and Emacs is @ 100% CPU and has been stuck there for minutes.

I'm going to try your profiler suggestion earlier and report back if I discover anything!

[saumets]

@hlissner tried the profiler method you described here (using ivy-prescient):
#774 (comment)

CPU jumped to 100% again so I had to terminate it with C-g.

This is what I got in the report:

Does that help?

[hlissner]

@hliebert counsel-fzf is superior for file search, since fzf is certain to be fast, though you do sacrifice a few aesthetic addons, like match highlighting for complicated queries and it requires you have fzf installed. In my testing, that kind of performance is only necessary for exceptionally large packages, so I'd rather not impose another dependency on users (or myself) if I don't have to.

@skalfyfan After some testing I've discovered there are inefficiencies in the counsel-projectile package (and prescient on top of it makes it worse).

I've come up with a decent solution which could use some more testing. Could you try adding this to your ~/.doom.d/config.el:

  (after! ivy
    (global-set-key [remap projectile-find-file] nil)
    (setf (alist-get 'projectile-find-file ivy-sort-functions-alist) nil)
    (setf (alist-get 'projectile-find-file ivy-re-builders-alist) 'ivy--regex-fuzzy)
    (after! ivy-prescient
      (add-to-list 'ivy-prescient-excluded-commands 'projectile-find-file nil #'eq)))

This will disable ivy-prescient and counsel-projectile features exclusively for projectile-find-file, which I've found to be significantly faster. I'm testing it on the linux kernel (with over 60k files) on an decent i5, ssd, 8gb ram linux pc and the latency, while noticeable, is acceptable, so I'm hoping this will resolve the issue for much smaller projects.

If you are in a project of 60k+ files or this fix doesn't help/is insufficient, I recommend either:

1. Using counsel-fzf instead. You'll need to install fzf, but the experience is guaranteed to be much faster.
2. Disabling +fuzzy support. Understandably, there is a learning curve to the default search syntax (it's the same as helm's when its fuzzy support is disabled), but I find it to be just as powerful, if not more so.

Let me know!

[hlissner]

I've pushed these settings upstream and then some. Let me know if that fixes your speed issues.

[saumets]

@hlissner thanks for trying but sorry to say the refactoring did not help. 😢 My (ivy +fuzzy) still starts locking my machine and CPU jumps to 100% when trying to find a file through projectile.

I guess I will have to stick with (helm +fuzzy) for now (which is fine by me). Let me know if there is any more debugging / profiling info that you'd like from me. Glad to help out if you want to dig further...!

[hlissner]

@skalfyfan This is very strange. I'm using this implementation on a 60k file project and it's only mildly sluggish (even on my core 2 duo toaster running El Capitan). A ~400 file project should be nothing. Even if you mistyped and meant 4k, it should still be snappy.

Would you mind trying these few tests for me?

1. Open any file in this problem project and run M-x eval-expression RET (projectile-get-ext-command); what is it's return value?
2. Run M-x doom/info and pastebin the result for me to see.
3. Try invoking projectile-find-file directly: M-x projectile-find-file (of course, with +fuzzy enabled)
4. Do you get this slowness in smaller projects? Say, projects of 10-20 files? If not, could you try profiling it with M-x doom/toggle-profiler (run it before the test, then again to produce the report).
5. Try disabling ivy-prescient entirely by adding (package! ivy-prescient :disable t) to your ~/.doom.d/packages.el file and running make. Then see if that resolves your issue.

[saumets]

@hlissner Does a01aeb6 mark the end of this issue? 🤔

If so we can close issue.

[saumets]

Closing this because @hlissner has officially moved the default completion mode to Helm now as identified by this commit: a01aeb6

[hlissner]

@skalfyfan Sorry for the late reply. I'm still uncertain about your specific case.

It is true that there are unavoidable limitations to ivy that would cause this, but again, we're talking about large projects of 30k+ files. A 500-file project should not be experiencing these slowdowns.

Your shell environment is suspect. Could there be any bottlenecks in your shell config? Perhaps initializing of plugins like oh-my-zsh, zprezto, pyenv, rvm, nvm, etc. I ask because, every time counsel (or helm, for that matter) calls ag/rg/pt (and it does so on every keystroke), it instantiates your shell environment, along with all its setup.

[hlissner]

@skalfyfan I've made some headway in optimizing ivy; specifically, I delay the garbage collector while the minibuffer is active. I hope this address your issue. If you have the time to update and hop over to ivy to try it, please let me know how it fares.

[saumets]

@hlissner will give it a go sometime over this weekend and let you know! Thanks! 👍