Tracking Down A Memory Leak

[psbrandt]

Hi,

I have a long running service that creates (and destroys) many Deno runtimes over its lifespan. The service has slowly been leaking memory over time and I've been trying to investigate what the cause might be. What I've discovered is that an application that contains only the following code seems to leak memory:

fn main() {
    deno_core::JsRuntime::new(deno_core::RuntimeOptions::default());
}

I believe this to be case because I get the following when running DHAT:

$ valgrind --tool=dhat --dhat-out-file=dhat-master.json ./target/debug/rusty-v8-memtest
==20430== DHAT, a dynamic heap analysis tool
==20430== Copyright (C) 2010-2018, and GNU GPL'd, by Mozilla Foundation
==20430== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==20430== Command: ./target/debug/rusty-v8-memtest
==20430==
==20430==
==20430== Total:     2,766,579 bytes in 3,128 blocks
==20430== At t-gmax: 664,075 bytes in 1,134 blocks
==20430== At t-end:  36,701 bytes in 659 blocks
==20430== Reads:     11,243,734 bytes
==20430== Writes:    3,642,207 bytes
==20430==
==20430== To view the resulting profile, open
==20430==   file:///usr/libexec/valgrind/dh_view.html
==20430== in a web browser, click on "Load...", and then select the file
==20430==   /opt/rusty-v8-memtest/dhat-master.json
==20430== The text at the bottom explains the abbreviations used in the output.

The full DHAT viewer output can be seen in my test repository here [PDF]. The raw DHAT JSON output is also available there.

It looked like the WASM engine was responsible for some of the leaked memory, so I did a custom build of V8 without WASM as follows:

$ GN_ARGS="v8_enable_webassembly=false" V8_FROM_SOURCE=1 cargo build -vv

Running DHAT then gives the following output:

$ valgrind --tool=dhat --dhat-out-file=dhat-no-wasm.json ./target/debug/rusty-v8-memtest
==29611== DHAT, a dynamic heap analysis tool
==29611== Copyright (C) 2010-2018, and GNU GPL'd, by Mozilla Foundation
==29611== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==29611== Command: ./target/debug/rusty-v8-memtest
==29611==
Error: unrecognized flag --wasm-test-streaming
The remaining arguments were ignored: --harmony-import-assertions --harmony-import-attributes --no-validate-asm --turbo_fast_api_calls --harmony-change-array-by-copy
Try --help for options
==29611==
==29611== Total:     3,628,306 bytes in 9,378 blocks
==29611== At t-gmax: 853,363 bytes in 1,294 blocks
==29611== At t-end:  18,789 bytes in 553 blocks
==29611== Reads:     34,087,294 bytes
==29611== Writes:    24,618,078 bytes
==29611==
==29611== To view the resulting profile, open
==29611==   file:///usr/libexec/valgrind/dh_view.html
==29611== in a web browser, click on "Load...", and then select the file
==29611==   /opt/rusty-v8-memtest/dhat-no-wasm.json
==29611== The text at the bottom explains the abbreviations used in the output.

DHAT viewer output here [PDF].

I also tried both of the tests above with jemallocator, but that didn't make a huge difference. See the results of those tests in my test repo.

Is my assumption correct that there should be no memory leaks (i.e. t-end should equal 0 in the DHAT output above) with the simple code I provided?

Am I doing something else wrong? Any tips on how I might be able to investigate further?

[nyannyacha]

Hi! @sbrandt 😋

I've once investigated the same thing.
The thought is that the deno_core::JsRuntime does not seem to cover the case spawning more than one isolate. If you look inside it, you'll see how it handles ExternalReferences or OpCtx, they just leak away, and they never get freed when the isolate is dropped.

Then, when you think about why it leaked, I can't shake the feeling that JsRuntime was designed only for sole isolation. If JsRuntime was designed without considering that the user can spawn many JsRuntime, they might think that the ExternalReferences or OpCtx might be okay to leak away.

I've seen once somewhere that a recent memory leak was related to EPT or pointer compression¹. However, I can't find anywhere a statement such as that the array of the external_references will be freed if the isolation drops. Looking at EPT, it sounds capable of garbage collecting, so I once thought something like the EPT could clean up the external_references, but I don't know how it works with the external references.

That's all I knew about that problem!
I hope this helps! 😁

Footnotes

1. https://bugs.chromium.org/p/v8/issues/detail?id=13640 ↩