Spyder very slow when working on large python files

[impact27]

Description

What steps will reproduce the problem?

A few things in spyder are slowing down spyder considerably, mostly whrn working with big files. The functions that take most of the time are:

• the paintEvent from spyder/plugins/editor/panels/scrollflag.py
  • Mostly computing the flags and adding all the painters
• __mark_occurences in spyder/plugins/editor/widgets/codeeditor.py

Versions

• Spyder version: 5.0.0.dev0 acb3dbf
• Python version: 3.7.5
• Qt version: 5.12.6
• PyQt5 version: 5.12.3
• Operating System: Darwin 19.0.0

Dependencies

cloudpickle >=0.5.0          :  1.2.2 (OK)
pygments >=2.0               :  2.4.2 (OK)
qtconsole >=4.6.0            :  4.7.0.dev0 (OK)
nbconvert >=4.0              :  5.6.1 (OK)
sphinx >=0.6.6               :  2.2.1 (OK)
pylint >=0.25                :  2.4.4 (OK)
psutil >=0.3                 :  5.6.5 (OK)
qtawesome >=0.5.7            :  0.6.0 (OK)
qtpy >=1.5.0                 :  1.9.0 (OK)
pickleshare >=0.4            :  0.7.5 (OK)
zmq >=17                     :  18.1.1 (OK)
chardet >=2.0.0              :  3.0.4 (OK)
numpydoc >=0.6.0             :  0.9.1 (OK)
spyder_kernels >=1.8.1;<2.0.0:  1.9.0.dev0 (OK)
qdarkstyle >=2.7             :  2.7 (OK)
atomicwrites >=1.2.0         :  1.3.0 (OK)
diff_match_patch >=20181111  :  20181111 (OK)
intervaltree                 :  None (OK)
watchdog                     :  None (OK)
keyring                      :  None (OK)
pexpect >=4.4.0              :  4.7.0 (OK)
sympy >=0.7.3                :  1.4 (OK)
cython >=0.21                :  None (NOK)
IPython >=4.0                :  7.11.0.dev (OK)
matplotlib >=2.0.0           :  3.1.2 (OK)
pandas >=0.13.1              :  0.25.3 (OK)
numpy >=1.7                  :  1.17.4 (OK)
scipy >=0.17.0               :  1.3.3 (OK)
pyls >=0.31.2;<0.32.0        :  0.31.1 (NOK)
applaunchservices >=0.1.7    :  0.2.1 (OK)

[impact27]

An example: (rename to .py)

pdc_lib copy.py.txt

[ccordoba12]

Could you help us with this one @impact27?

[jnsebgosselin]

I think I can help with optimizing the scrollflag paintEvent if you need help with it.

[ccordoba12]

That'd be really appreciated, thanks!

[jnsebgosselin]

That'd be really appreciated, thanks!

Ok I'm on it. I won't touch the __mark_occurences in spyder/plugins/editor/widgets/codeeditor.py though.

[bcolsen]

This would be related to #10992

[impact27]

I opened #11036 for __mark_occurences
I go from taking 33s for __mark_occurences to 1.2s (mostly taken by __find_next)
Also process_code_analysis goes from 3.7 to 0.2s

[mluerig]

so this issue still persists for me, even when deactivating code folding, and it only stops when I deactivate the LSP altogether. symptoms are that whenever I add or modify code to large files, my input is being evaluated, and the the editor freezes for about 2 secs before I can do something again.

do you want me to open a new issue?

[ccordoba12]

Hey @mluerig, we're aware of this problem (see issue #21637). Some changes to improve the situation will arrive in Spyder 6.0 and we'll continue working on it in 6.1 (to be released at the end of the year).

To give you more context about this: it's not an easy problem to solve because we're not able to run the most expensive computations needed for the LSP in threads due to Python's GIL (Global Interpreter Lock). So we have to find clever ways to do that and that takes time and effort.

[mluerig]

hey, thanks for the update - ok, I'll be patient 😄