Spyder 4.0.0 unbearably slow when adding or removing lines

[gabrielclow]

Problem Description

Spyder is very very slow everytime I add (with Enter) or remove (with shortcut set to Ctrl+D) lines in the Editor. It takes on average something close to a second for every line. When holding, the line commands will buffer and then they will be added or deleted as slow as maybe two seconds per line.

Same issue after spyder --reset. Also tried toggling auto completion. This has been happening since the beta/release candidate and I havent been using it because of this. Spyder is unusable now like this for me.

What steps reproduce the problem?

1. Press Enter or Ctrl+D

What is the expected output? What do you see instead?

I expect it to not take so long like older versions.

Versions

• Spyder version: 4.0.0
• Python version: 3.7.5
• Qt version: 5.9.6
• PyQt version: 5.9.2
• Operating System name/version: Win 10

Dependencies

cloudpickle >=0.5.0 : 1.2.2 (OK)
pygments >=2.0 : 2.5.2 (OK)
qtconsole >=4.6.0 : 4.6.0 (OK)
nbconvert >=4.0 : 5.6.1 (OK)
sphinx >=0.6.6 : 2.2.2 (OK)
pylint >=0.25 : 2.4.4 (OK)
psutil >=0.3 : 5.6.7 (OK)
qtawesome >=0.5.7 : 0.6.0 (OK)
qtpy >=1.5.0 : 1.9.0 (OK)
pickleshare >=0.4 : 0.7.5 (OK)
zmq >=17 : 18.1.0 (OK)
chardet >=2.0.0 : 3.0.4 (OK)
numpydoc >=0.6.0 : 0.9.1 (OK)
spyder_kernels >=1.8.1;<2.0.0: 1.8.1 (OK)
qdarkstyle >=2.7 : 2.7 (OK)
atomicwrites >=1.2.0 : 1.3.0 (OK)
diff_match_patch >=20181111 : 20181111 (OK)
intervaltree : None (OK)
watchdog : None (OK)
keyring : None (OK)
pexpect >=4.4.0 : 4.7.0 (OK)
pympler : None (OK)
sympy >=0.7.3 : 1.4 (OK)
cython >=0.21 : 0.29.14 (OK)
IPython >=4.0 : 7.10.1 (OK)
matplotlib >=2.0.0 : 3.1.1 (OK)
pandas >=0.13.1 : 0.25.3 (OK)
numpy >=1.7 : 1.17.4 (OK)
scipy >=0.17.0 : 1.3.1 (OK)
pyls >=0.31.2;<0.32.0 : 0.31.2 (OK)
rtree >=0.8.3 : 0.8.3 (OK)

[ccordoba12]

@gabrielclow, your description is not enough. Please post:

1. What kind of files are you editing (Python or other)?
2. How many lines you file have?
3. Do you have things like "Indent guides" enabled?

Finally, please post a video (animated gif with the Licecap) problem to see what you're exactly referring to.

[gabrielclow]

@gabrielclow, your description is not enough. Please post:

1. What kind of files are you editing (Python or other)?
2. How many lines you file have?
3. Do you have things like "Indent guides" enabled?

Finally, please post a video (animated gif with the Licecap) problem to see what you're exactly referring to.

1. I'm editing only .py files
2. Doesn't look like it depends on the number of lines. Happens in small files with 50 lines to big ones with 5k lines.
3. I only use auto-complete (config in screenshot). The rest is turned off (no linting, no code style linting, no docstring linting, no indent guides). I have no other languages set-up and I also tried toggling Kite on and off.

In the gif, I hold Enter, wait for the new lines to be created and then hold Ctrl+Z to remove them (the same effect as Ctrl+D, so I'm guessing it also doesn't have anything to do with the shortcut). Spyder then stops and adds or deletes the lines very slowly instead of doing it smoothly:

update: apparently, if I close some files the slow down is less noticeable!
update 2: also, the amount of slowing down is apparently related to the number of open files + number of lines in each file, but it happens regardless of the one currently selected. If I keep only a few smaller files and close the rest, it's much smoother
update 3: if I hold down too long or press too fast, Spyder can even hang for a while

[ccordoba12]

also, the amount of slowing down is apparently related to the number of open files

How many files do you have opened when you saw the worst slowdown?

[gabrielclow]

also, the amount of slowing down is apparently related to the number of open files

How many files do you have opened when you saw the worst slowdown?

I mostly work with 6-8 files open. Usually about two of them are bigger (4k-5k lines) and the rest are smaller files with 100-500 lines.

[ccordoba12]

@CAM-Gerlach, have you seen something like this before?

[1313e]

If I may comment on this, I am having the same problem as well, which actually made me uninstall Spyder 4 and go back to Spyder 3.3.6.
I have only the settings enabled that are enabled in Spyder 3 as well, so PEP8 checking; code completion and details are enabled; but no docstring checking; code snippets; hover descriptions; kite; etc.
The slowdown is as shown in @gabrielclow's GIF, although it can sometimes take as much as 2 seconds for a single line to be inserted.

As I build large packages, it is not uncommon for me to have like 30-50 Python files open simultaneously, some of which can easily contain over 3k lines of code (I had these open when I was noticing the massive slowdown).
Since I use 3 monitors simultaneously to edit my scripts most of the time, which can hold a maximum of 8 panels, and I need to switch between files frequently, closing most of these files will not work for me.

I personally get the feeling that the main reason for this slowdown, is that there are quite a few databases and scripts that need to be updated/ran whenever an open file is modified.

[1313e]

Is there, by any chance, a script that is ran on all lines in all open files whenever a file is modified?
It would explain why the slowdown seems to scale with the combined total number of lines in all open files, and why it seems to be independent of the file being edited.

[bcolsen]

I can confirm this bug. It's not so bad(didn't really notice until now) on my computer but adding/removing line is much slower than adding text. The enter problem is really noticeable if you hold it down.

[ccordoba12]

@bcolsen, how can we reproduce it?

[bcolsen]

Open about 40 files and it seems to slow pretty good. It looks like there might be something in docdidchange that is checking all the files not just the active one. Maybe the outliner.

[CAM-Gerlach]

@ccordoba12 As far as I can tell, this behavior is basically the same as #8864 , and sure enough I can repro it with clean settings on Spyder 4 final with just mainwindow.py open so long as indent guides are enabled (having split panes makes the lag double or more). Deleting lines has the same major lag (exactly as described here; with a ~1s latency and a ~1s repeat delay on a moderately powerful machine, and scales proportionally to file length and the number of split panes open) as the other things I tested there (moving lines, duplicating lines, etc), and I can confirm it still happens on Spyder 4 final with only the temp file and mainwindow.py open with clean settings.

However, I thought indent guides were not enabled by default, and indeed they don't appear to be enabled after I reset prefs, but I can't seem to repro anywhere near this amount of lag with any other setting (real-time code analysis, OTF completion, underline errors and warnings, show blank spaces, etc,); though with all of them enabled save indent guides it does lag quite a bit.

Is there, by any chance, a script that is ran on all lines in all open files whenever a file is modified?

LSP gets called, and I initially though that was it but based on its behavior with split panes, specific displayed items but not others, etc. I'm thinking it may be redraw-related instead (drawing the whole file instead of just the visible area, etc). But that's way above my pay grade.

[1313e]

@ccordoba12 As far as I can tell, this behavior is basically the same as #8864 , and sure enough I can repro it with clean settings on Spyder 4 final with just mainwindow.py open so long as indent guides are enabled (having split panes makes the lag double or more).

If I also enable the indent guides together with having my 50 files open and 8 split panes (which is how I usually edit my files), the lag increases to anywhere between 6 to 10 seconds per new line.
The computer I am using is a high-end gaming desktop, so it is not really because of the power of the machine.

[ccordoba12]

@dalthviz and I tried to reproduce #8864 unsuccessfully. So sorry to say, but until we're able to reproduce it, there's nothing we can do about it.

[ccordoba12]

Open about 40 files and it seems to slow pretty good. It looks like there might be something in docdidchange that is checking all the files not just the active one. Maybe the outliner.

Ok, that's a good start. We'll check this one out.

[CAM-Gerlach]

@ccordoba12 I've figured out the source of this lag, which is in fact appears to be the main "baseline" lag felt in Spyder 4 vs. Spyder 3 that I've talked about for a while: As @bcolsen suggested, it is indeed the outline pane, although it actually occurs regardless of whether said pane is open or closed. On a lower-end machine, it is actually visible (though nowhere near as bad) with only temp.py and mainwindow.py open, nothing else enabled and no outline pane open. Here's how to repro from clean settings:

1. Open mainwindow.py and a split editor panel, and set the right panel to show temp.py and the left to show mainwindow.
2. Hold down delete/duplicate/move line in the left panel (mainwindow), undo/redo, etc. for the same. A small but noticeable lag is apparent.
3. Open the Outline pane (which was hidden by default). This (due to a minor but useful apparent bug) de-syncs the file shown in the outline pane from the one focused, due to it switching to show the file active in the right-most split panel rather than the active split panel when opened/unhidden.
4. Retry deleting/etc/ lines in mainwindow. Very little lag is present and repeat rate is higher (likely will be more noticeable with lower performance machines)
5. Click on right panel (temp.py), and then back to left panel (mainwindow) to make the outline explorer properly show the outline of the left panel.
6. Retry deleting/etc., and the lag is back, same as in step 2 (when outline explorer wasn't visible at all)

To test it with many files, I searched import os in the spyder directory, opened the first 40-50 files in a split panel view (fewer larger files should produce the equivalent effect) and repeated steps 2-6 above. Even on a short file, there was a far larger amount of lag (similar to indent guide level on Mainwindow) when deleting lines in both steps 2 and 6 (outline explorer hidden, and shown on the active file), while there was essentially no lag (as if no files were open, or Spyder 3 level) on step 4 (outline explorer open and de-synced from active file).

Meanwhile, enabling indent guides for both times, deleting/etc lines was very slow regardless of the number of files open, and the resulting lag appeared to only be substantially affected by the length of the current file being modified and the number of split panes showing the file.

[gabrielclow]

3. Open the Outline pane (which was hidden by default). This (due to a minor but useful apparent bug) de-syncs the file shown in the outline pane from the one focused, due to it switching to show the file active in the right-most split panel rather than the active split panel when opened/unhidden.

This really does sound like what I'm experiencing but just adding another detail: I work with one separate editor window on another display and I close the editor pane on Spyder everytime I launch it. So I still experience this bug with a single editor window without any split panes, if that's relevant. Indent guides also make it take much longer.

I hope you can find the answer and this gets fixed soon so I can use the great dark mode in 4.0.0

[bcolsen]

I can confirm that every time a line is added or removed the outliner updates all of the files that are open. This leads to increases in time for line removal /addition that increases with the number of files open. I have made a simple "fix" in the above PR that seems to remedy the problem with many open files. @impact27 Thoughts?

A separate but obviously associated issue is that large files (>2000 lines) are also slow to update with line changes. Having 4 or 5 of these large files open currently make things even slower with update_all makes. With the above patch, the problem that @CAM-Gerlach is describing above will still happen on the large file but the small file should always be fast.

[goanpeca]

A separate but obviously associated issue is that large files (>2000 lines) are also slow to update with line changes. Having 4 or 5 of these large files open currently make things even slower with update_all makes. With the above patch, the problem that @CAM-Gerlach is describing above will still happen on the large file but the small file should always be fast.

It seems we need to move the UI blocking process to a thread.

[bcolsen]

It seems we need to move the UI blocking process to a thread.

This would be best. I think it's associated with the syntax highlighter at that point however.

[goanpeca]

I think it's associated with the syntax highlighter at that point however.

Yeah that is a problem since the syntax highlighter infrastructure of Qt runs on the main thread :-\

[CAM-Gerlach]

@bcolsen In addition to only updating the current file, it should also not update if the outline explorer isn't even visible (as my testing confirms it does now).

[Vaderv]

I am having the same issue.
Spyder has become extremely slow.
I hope if there are any packages I need to update.

[CAM-Gerlach]

@NaderNazemi As you can see from the issue, we've performed comprehensive testing, isolated the causes of the issue and already implemented a fix that should mostly resolve it in Spyder 4.0.1 that will be released very soon, with more improvements on the way. Please be patient, or if you're an advanced user you can test the latest Spyder development branch from Github to try out the fix yourself. Thanks.

[DGuidi]

I see it's fixed, but in my 4.1.4 version (probably there are newer versions out there) the problem happens when Kite autocompletion app is closed.

[CAM-Gerlach]

Hey @DGuidi , thanks for letting us know. Just FYI, Spyder 4.1.5 was released a few weeks ago with some minor further improvements, and several larger changes to hopefully improve Editor performance when scrolling, typing and using real-time code analysis/indent guides/etc. are either implemented for the next version or are in testing right now. If you're feeling adventurous, it would be really helpful if you could give #13864 a try and see if it improves things for you; I'll be testing it shortly as well.

As for Kite, I don't use it and its a third-party proprietary plugin, but it sounds like you've isolated a specific case where its slow which is really helpful to solving it, so hopefully one of their developers, or someone else so inclined, will be able to figure out what's going on. Best of luck!

[DGuidi]

@CAM-Gerlach and all, just to add some info, I'm using WinPython64-3.8.5.0cod from https://winpython.github.io/ so maybe the problem relies on some configuration in this package or maybe in my windows machine.
Hard to me to upgrade the spyder version, at least until winpython is updated: I don't have the ability to install software in my machine.

[DGuidi]

Disabling kite completion and also fallback completions NOT works for me :( 

[CAM-Gerlach]

@stonebig Any chance you might know anything about this?

For the time being you could try disabling any options under the Source menu, Preferences > Editor and Preferences > Completion and Introspection that you don't particularly need, particularly indent guides which can cause significant lag. Other than that, we have a number of fixes ready or in final testing for Spyder 4.2.0, to be released in the next few weeks, that aim to improve Editor performance and resolve most of the causes of slowdown, particularly on larger files, but without being able to test those to confirm they improve things for you, I'm not really sure what else to recommend, sorry.

[stonebig]

Hi @CAM-Gerlach , 95% of time Windows is dead slow, it's related to anti-virus activity or memory saturation.

Also, Spyder optional plugins may not improve the situation

[CAM-Gerlach]

Okay, thanks. Hopefully 4.2.0 will resolve, or at least greatly improve the situation for @DGuidi

[dartoon]

I have noted this similar issue when I use Anaconda spyder version = 4.1.2. I find that this happens when I have external monitors connecting to my mac. However, the lagging only happens on my mac screen and it will disappear when the spyder editor is moved to the external monitor. This is really strange.

[CAM-Gerlach]

Hello @dartoon ; sorry you're running into slowness. Just FYI, the problem in this issue was resolved in Spyder 4.0.1, there has been, however, a slowness issue reported in macOS Big Sur if you're using that version, see e.g. #14735 for more on that and an easy workaround. Furthermore, Spyder 4.1.2 is a year out of date, with 4.2.5 the most recent version in the 4.x series, and Spyder 5.0.0 just released as well, and numerous performance and bug-related issues have been fixed since then, so please try 4.2.5 using our recommended macOS-native DMG installers to determine if your problem is solved there. Thanks!