Memory leak when redefining module

[Gilga]

Happens in both versions of Julia:

1.0.0

julia> versioninfo()
Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-4510U CPU @ 2.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, haswell)

1.1.0

julia> versioninfo()
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-4510U CPU @ 2.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, haswell)

Do the following Tests and watch your memory

1. Test (Works fine in Julia 1.0.0 Windows, but not in Julia 1.1.0)

julia> module TEST; x=zeros(Float32,128^3*300); end; # ca. 2.5 GB memory
julia> module TEST; end;
julia> GC.gc() # clean up fails?

2. Test (Fails in Julia 1.0.0 and Julia 1.1.0)

julia> module TEST; x=zeros(Float32,128^3*300); f() = nothing; end; # ca. 2.5 GB memory
julia> module TEST; end;
julia> GC.gc() # clean up fails?

Note

• My intention is to do a full clear/clean of a module. Removing all variables, constants, functions etc.
• Maybe Related:
  • Memory leak with function redefinition
  • Segmentation fault from changing constants
• Any thoughts on this?

[chethega]

Can reproduce. Works fine on julia 1.0.3, fails on 1.2.0-DEV.219. I tested on x86_64 linux with glibc 2.28.

Procedure: Repeatedly paste the following line into the REPL:
julia> GC.gc(); GC.gc(); module TEST; x=zeros(Float32,128^3*100) end; GC.gc(); GC.gc();

On 1.0.3: julia consumes some memory and sometimes releases memory back to the OS. Regardless, this can be repeated arbitrarily often.
On master: Eventually I get an out-of-memory error.

I used this procedure to disambiguate between memory leaks (memory does not get re-used even under subsequent memory pressure) and questionable decisions about returning memory to the OS (there where some issues 30888, 25884 on linux, possibly caused by glibc malloc somehow deciding to grow the heap instead of mmapping some large allocations, and then being unable to shrink the heap afterwards, due to contiguity; this memory is not leaked and is still used for subsequent work, it just doesn't get returned to the OS).

[Gilga]

@chethega Did you do the second test for julia 1.0.3 too? When there are functions present or even links to modules by the using statement, this memory problem occurs too.

julia> module TEST; x=zeros(Float32,128^3*300); f() = nothing; end; # ca. 2.5 GB memory
module TEST; end; # redefine module (reset to blank module)
GC.gc() # clean up fails?

For me varinfo() and names() clearly shows clean module (no variable x symbol present) after use of GC. So why is this memory still being used and where? I can manage to free this memory manually when i use clear! on variable x (or set it to nothing) before redefining the module and GC (so this still works). I am hesitant to free the memory manually for modules, because you might step on more issues like memory release for constants, nested modules, etc.

[vtjnash]

Not a bug, since it prints an error also (WARNING: replacing module TEST.)