New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak with %matplotlib inline #7270
Comments
I hit this bug as well, is there any way to get inline plots without memory leaks? I do not want to launch separate processes for each plot, since the arrays are quite large. |
Can you check this when memory usage increases: len(IPython.kernel.zmq.pylab.backend_inline.show._to_draw) That's a list where figures are being stored. They should be there only temporarily, but maybe they're building up without getting cleared. |
len(IPython.kernel.zmq.pylab.backend_inline.show._to_draw)=0 BTW, I'm plotting using |
OK, so much for that theory. It's possible pandas keeps some data around plots internally as well. The original report doesn't involve pandas, though. How much memory does each additional plot appear to add? |
ok, this seems to be my case, I was using pandas 0.16.0, but the issue is fixed in master: |
Great, thanks. Leaving open since the original report didn't involve pandas. |
This can be reproduced more simply: import matplotlib
import matplotlib.pyplot as plt
import matplotlib.ticker
%matplotlib inline
import os
import sys
import StringIO
import urllib, base64
from matplotlib import rcParams
rcParams['figure.figsize'] = (24, 6)
rcParams['figure.dpi'] = 150
def friendlyPlot():
fig, ax = plt.subplots()
ax.plot(range(1000))
fig.savefig('tmp.png')
plt.close('all')
for i in range(500):
friendlyPlot() This does not leak memory so it is something on the IPython side not the pyplot side (I think). import matplotlib
matplotlib.use('agg')
import matplotlib.pyplot as plt
import matplotlib.ticker
import os
import sys
import StringIO
import urllib, base64
from matplotlib import rcParams
rcParams['figure.figsize'] = (24, 6)
rcParams['figure.dpi'] = 150
def friendlyPlot():
fig, ax = plt.subplots()
ax.plot(range(1000))
fig.savefig('tmp.png')
plt.close('all')
for i in range(500):
friendlyPlot() |
@tacaswell With your test code IPython on Windows 7 consumes here approximately 1.7GB which are not freed afterwards. Running with a slightly higher number of iterations leads to a memory error. So this is still an issue. |
@asteppke The first or second block? |
@tacaswell With your first test code ( Both tests are executed with Python 3.4 and IPython notebook version 4.0.5. |
I've played with this a bit more. I notice that if I re-run the for loop in @tacaswell's example a few times, memory usage doesn't increase - it seems to be the number you create in a single cell that matters. IPython certainly keeps a list of all the figures generated in the cell for the inline backend, but that list is quite definitely being cleared after the cell runs, which doesn't make memory usage drop, even after doing Could our code be interacting badly with something in matplotlib? I thought I tried grabbing a reference to one of the figures and calling |
I'm dropping this to milestone 'wishlist'. We want to fix it, but at the moment we're not sure how to make further progress in identifying the bug, and I don't think it's worth holding up releases for it. Anyone who can make progress gets brownie points. Also cake. |
Not really progress, but the memory seems to be lost somewhere inside the kernel. Neither does calling |
I also wondered if it was creating uncollectable objects, but those should end up in I think someone who knows about these things is going to have to use C-level tools to track down what memory is not getting freed. There's no evidence of extra Python objects being kept alive anywhere we can find. |
I'll second that a fix on this issue would be appreciated. |
We know, but at present no-one has worked out the cause of the bug. |
+1 |
1 similar comment
+1 |
BTW, I'm still hitting this issue from time to time on latest matplotlib, pandas, jupyter, ipython. If anyone knows any debugger that can help to troubleshoot this multi-process communication, then please let me know. |
Could it perhaps have anything to do with the browser cache mechanism? |
Good thought, but I don't think so. It's IPython's process taking up extra memory, not the browser, and |
Hi, I believe I have found part of the culprit and a way to significantly, but not completely, reduce this problem! After scrolling through the Here is the code to verify this (based on @tacaswell's snippet above), useful for anyone trying to implement a fix. Initialization: import matplotlib
import matplotlib.pyplot as plt
import matplotlib.ticker
%matplotlib inline
matplotlib.rcParams['figure.figsize'] = (24, 6)
matplotlib.rcParams['figure.dpi'] = 150
from resource import getrusage
from resource import RUSAGE_SELF
def friendlyPlot():
fig, ax = plt.subplots()
ax.plot(range(1000))
fig.savefig('tmp.png')
plt.close('all') Actual test: print("before any: {:7d} kB".format(getrusage(RUSAGE_SELF).ru_maxrss))
friendlyPlot()
print("before loop: {:7d} kB".format(getrusage(RUSAGE_SELF).ru_maxrss))
for i in range(50):
friendlyPlot()
print("after loop: {:7d} kB".format(getrusage(RUSAGE_SELF).ru_maxrss))
import gc ; gc.collect(2)
print("after gc: {:7d} kB".format(getrusage(RUSAGE_SELF).ru_maxrss)) Running it for 50 iterations of the loop, I get:
Running it for 200 iterations of the loop, I get:
which shows the almost linear increase in memory with iterations. Now to the fix/workaround: call With 50 iterations:
And with 200 iterations:
Which confirms that only a constant increase (independent of iterations) is left. Using these numbers, I make a rough estimate of the leak size per iteration:
And for a single iteration of the loop, I get Also, strangely, running a small-scale test (only few iterations) repeatedly in the same cell without restarting the kernel is much less consistent, I have not been able to understand this or determine a pattern. I hope someone with more knowledge of the internals can take it from here, as I lack the time and knowledge to dive deeper into it right now. |
it works |
Hey everyone
I've found a problem. Just launch the code and look at the memory. Then delete "%matplotlib inline" and launch again.
The text was updated successfully, but these errors were encountered: