detect_equilibration_binary_search is not memory safe

[smcantab]

today my cluster administrator has shut down 2 nodes because my jobs using detect_equilibration_binary_search had filled 100% of the memory and were swapping dangerously with the disk. I spent the afternoon looking into it and I have narrowed down the problem to the implementation of np.fft which is called by statsmodel through sm.tsa.stattools.acf.

Turns out that "fft stores a cache of working memory for different sizes of fft's, so you can run into memory problems if you call this too many times with too many different n's", which is exactly what we do during the binary search.

I have added my 2 cents to an issue that I found in the numpy github repo. My solution at the moment is to add a call to

np.fft.fftpack._fft_cache.clear()

which clears the global variable _fft_cache after each for loop or after each iteration. Would you be interested in a PR that addresses this issue?

[kyleabeauchamp]

I think that sounds like a reasonable workaround.

[smcantab]

it is necessary to clear the cache after every fft call, so I have added the necessary lines to the statisticalInefficiency_fft function in PR #169

[smcantab]

P.S. tests pass locally

[kyleabeauchamp]

Thanks!

[smcantab]

Thanks for merging!

On Fri, Jan 30, 2015 at 3:17 PM, Kyle Beauchamp notifications@github.com
wrote:

Thanks!

—
Reply to this email directly or view it on GitHub
#168 (comment).