-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak while having spyder run too much stuff or for too long. #14792
Comments
Hi @leet0rz, Can you share with me more information about what things you had opened in Spyder to see if we can reproduce this issue? |
Window 1: 40 lines of code Just basic, simple code that I did not even run, nothing major. It'll also happen if I just open a regular txt file with notes of mine. I am sure the same would happen if I open absolutely nothing and kept it running for a few days. If you do this and hold enter creating new lines it'll start lagging and chopping its way while making new lines. |
By windows you mean editor tabs @leet0rz? Also, could you post a screenshot of your current setup so we can take a look at it? Thanks! |
Yeah windows as in tabs. Here is a screenshot currently, 4 tabs open:
https://i.imgur.com/6iYb1My.png
…On Thu, Feb 25, 2021 at 6:30 PM Carlos Cordoba ***@***.***> wrote:
By windows you mean editor tabs @leet0rz <https://github.com/leet0rz>?
Also, could you post a screenshot of your current setup so we can take a
look at it? Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#14792 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AG6S3GD3AJ7V6H53MWU5YGDTA2CLDANCNFSM4X66D2KQ>
.
|
I got the very same problem. |
@MuellerSeb, most of these problems should be fixed in our latest version (5.2.1). |
I got 5.2.1 installed. Only extension I have installed is the spyder-terminal.
fills up my RAM with 5MB / sec while doing nothing. This is 300MB / min. After half an hour with spyder in the background, and other stuff open my system begins to stutter, since there are 10GB of ram allocated by this process. So I think, this is not solved. |
I tried: pip uninstall spyder-terminal
pip install -I --no-cache-dir spyder But still, I open spyder and can watch the memory being filled by the same process. |
There has to be something else going on not related to Spyder because I've had a Spyder instance open for two weeks on Linux and I haven't seen any memory leak. Given this line
it seems you installed Spyder with pip in your system Python. If that's the case, you probably broke it and now you have to reinstall your distro, or at least remove everything under After that, please try to use Spyder with Anaconda/Miniconda, or to install it with pip in a virtualenv. |
@leet0rz @ccordoba12 I can reproduce what seems to be spyder memory leak on windows. Spyder version is 5.2.2. I did the following:
memory report:
A difference with the issue as described above is that for me the system does not really slow down, but at some point I get out of memory errors. |
@ccordoba12 This still happens with Spyder 5.3 (Windows). |
According to the Python docs,
and
So it seems a very low level feature that doesn't happen directly in Spyder (hence not something we can fix here) but in spyder-kernels. My guess is that that's related to the way IPython handles its several namespaces and probably not something that can be fixed, sorry |
@peendebak Yeah, I just started using vscode instead. Tons of other editors use python and do not have these memory leak issues meanwhile Spyder does for some reason, and it should not have these issue in this day and age. Maybe it's base is broken with some broken code, I dont know but other editors manage to work without leaks. |
@leet0rz, most leaks should be fixed now in our latest version (5.3.0). If you're using an older one, please update and try again. |
@leet0rz @ccordoba12 I used heapy to track down some of the objects. I can now reproduce the increasing number of cell items in plain python as well. In spyder itself (or just a plain spyder-kernel) I see no additional memory leakage (tested spyder 5.3 on python 3.9). I will see whether I can pinpoint the issue, but it indeed does not seem related (directly) to spyder. |
@ccordoba12 I'm using Spyder 5.3.0 on Ubuntu, installed via conda, and have something similar. I was training a few different models for a couple days straight (different consoles, using TF/Keras which print the progress to the console and graphs to the plots pane). It filled my RAM and started eating away at my swap space. When the trainings were done, I closed all of the consoles which recovered about half of my memory, but the spyder process itself was still using 55GB RAM (yes, fifty five). I opened the internal console to poke around and try to find what might be using the memory, but accidentally killed it 😆 don't call spy.window.destroy() Sometimes, running three or four consoles that are all training will end up slowing down Spyder to an unusable state, even if only half of my RAM is in use. Switching between tabs is painfully slow (ie, a minute), the progress stops updating in the console pane, and this is usually precipitated by me trying to open a new console while those others are running. (This is a recurring issue, so I usually just restart spyder every few days). I've tried running in debug mode, but the logs grow at tens of MB a minute so I couldn't let it run long enough to really see what's going on. Any suggestions for how I can find what's eating the memory? It may be Spyder related since I had killed all of the spyder_kernels.console processes (except the new console that was created after closing them all) And your quote was about de-referencing, not removing, which I'm interpreting as C pointer de-referencing, no? And if it is an issue with cell objects, they do have a dealloc, but I don't know anything about how any of that works. https://github.com/python/cpython/blob/main/Objects/cellobject.c#L161 |
This could have to do with the Variable Explorer consuming too much memory. To check that, please go to the menu
and disable it |
Awesome thanks for the suggestion, I'll let you know! |
@ccordoba12 the variable explorer didn't seem to change anything. I played a bit with what peendebak said above, looking at all the objects gc can list. It seems like a huge part of the problem is the large amount of text that my script generates (I turned off the progress bars and that seems to have made a huge impact). However, doing something as simple as
does NOT replicate it, so it must be something more complicated. Even after closing the consoles, something else still must be referencing these objects (spyder?). For example, there are still references to matplotlib pngs generated during my training:
Any thoughts on how I could release these references? After closing the console, or removing all plots, I'd expect them to be freed from memory. For example, |
I think you hit the nail on the head! That probably means that you're experiencing the bug reported on issue #12847. Please read it for some workarounds. |
Hi @ccordoba12, I'm facing a similar problem, but I think I found its cause and a possible solution. My problem is that Spyder slowly starts using up more and more of my RAM even though I'm not using Spyder for anything and it is left idle. Eventually, my entire PC starts to lag due to the excessive use of RAM. I tried the workaround in #12847, but this did not work. Specifically, I added
I'm also running Ubuntu 20.04.4, and I installed miniconda and not the full anaconda. I first installed Spyder in a new conda environment using the What solved my problem, however, was as follows:
UPDATE 1 UPDATE 2 $ spyder I kept getting this error Traceback (most recent call last):
File "/home/mahmedta/anaconda3/envs/openglue/lib/python3.10/site-packages/qtpy/QtWebEngineWidgets.py", line 21, in <module>
from PyQt5.QtWebEngineWidgets import QWebEnginePage
ModuleNotFoundError: No module named 'PyQt5.QtWebEngineWidgets'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mahmedta/anaconda3/envs/openglue/bin/spyder", line 11, in <module>
sys.exit(main())
File "/home/mahmedta/anaconda3/envs/openglue/lib/python3.10/site-packages/spyder/app/start.py", line 247, in main
from spyder.app import mainwindow
File "/home/mahmedta/anaconda3/envs/openglue/lib/python3.10/site-packages/spyder/app/mainwindow.py", line 57, in <module>
from qtpy import QtWebEngineWidgets # analysis:ignore
File "/home/mahmedta/anaconda3/envs/openglue/lib/python3.10/site-packages/qtpy/QtWebEngineWidgets.py", line 28, in <module>
from PyQt5.QtWebKitWidgets import QWebPage as QWebEnginePage
ModuleNotFoundError: No module named 'PyQt5.QtWebKitWidgets' To get around this error, I removed the current environment and created a new one. This time, I first ran $ conda install spyder which installed Spyder 5.1.5 from the anaconda channel. Then, I ran $ conda install -c conda-forge spyder=5.2.2 which installed version 5.2.2 of Spyder from the conda-forge channel. When I did this, Spyder started without the |
@ccordoba12 Thanks, but unfortunately it's definitely deeper than this. I haven't had the time to dig enough into the cause, but here's some more info... If I run a hyperparameter optimizing script that either:
option 1 definitely has a faster memory leak than option 2 (though both do have an issue). With option 2, I can see overall system memory usage increasing during the training, then dropping off when the training process is killed. After enough cycles though, I can see overall usage is slowly creeping up. When I have some free time, I'll try to figure out what's occupying most of the memory. |
@stevetracvc, that doesn't seem related to Spyder. I mean, perhaps memory is not released with option 1. because the objects used to do the hyperparameter optimization are retained in memory after it finishes. Could you try to run option 1. in a Jupyter notebook to check if the memory leak also shows up there? Since Spyder and Jupyter use the same architecture to run code, that would imply that your problem is not caused by Spyder. |
I'm having this issue too, I left it open running a few threads printing off data from a Binance endpoint and now running a simple print command on a different cell takes 10-20 seconds to show up in the console. If I restart, everything works fine |
For some reason the RAM usage just slowly climbs with Spyder running with the Graphics Backend set to "tkinter". I changed the Graphics Backend from "tkinter" to "Qt5" and this solved my memory leak problem. This is with Spyder 5.3.3 |
thanks for the suggestion but unfortunately it didn't seem to work for me. I was on inline before and now it feels that it's slightly slower. I wonder if it's just a performance thing. There are 3 threads running and when they are off the individual cell commands become responsive. I migrated this code from vscode and it had no issues there, maybe it's the way spider checks for new commands, when we hit run on individual threads while threads are active, and not actual lag. Because the code itself runs fine, just new commands in different cells take 10-15 seconds |
It's really a shame, the UI for this IDE, the individual cells, consolidated output, undock and customization and organizing features but the delays just keep getting worst. Going to go back to Vscode grudgingly, but I'll save this file as is if there is any way I can help 3 threads running Only thing I changed is changed New consoles to the working directory of the current console Spyder 5.42 |
@pearlmam, that's fixed in our latest version (5.4.2). So, please update.
@kian25, please upload it somewhere so we can take a look at it. That way we can try to reproduce your problem on our side and see how to fix it. |
I'm encountering a persistent memory problem, similar to one reported years ago in this issue, where the Symptoms:
This issue forces me to use Spyder solely for code editing and basic execution, while I resort to running scripts for model training and testing via the terminal to avoid crashes and data loss. Temporary Workaround: My environment has been exported with Any idea if this is related to the issues mentioned above, or yet another (separate) memory leak issue with Spyder? |
Description
What steps will reproduce the problem?
Just having a couple of windows open, leave the program running for 1-2 days, sometimes hours and it will slow down real bad to a point pressing enter for new lines takes a couple of seconds. Obviously unacceptable, back to VScode until you guys fix this.
Versions
Dependencies
The text was updated successfully, but these errors were encountered: