3.5.8 causes an OOM when used with @google-cloud/datastore

[scottbessler]

1. Is this a client library issue or a product issue?

This is an issue that occurs with a specific version (3.5.8) of google-gax.

2. Did someone already solve this?

no

3. Do you have a support contract?

no

Environment details

> uname -a
Linux f86fa67c0fcb 5.15.0-1030-aws #34~20.04.1-Ubuntu SMP Tue Jan 24 15:16:46 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
> node --version
v18.6.0
> npm --version
8.13.2
> yarn --version
3.3.1

google-gax: 3.5.8

Steps to reproduce

> cat test.js
var ds = require('@google-cloud/datastore');
var ds2 = new ds.Datastore({ namespace: 'record-properties-test' });
var root = ds2.key(['RecordProperty','Root']);
var q = ds2.createQuery('RecordProperty').hasAncestor(root);
ds2.runQuery(q).then(console.log).catch(console.error);

> node --max-old-space-size=512 test.js

<--- Last few GCs --->

[6481:0x6e52790]     7136 ms: Scavenge (reduce) 508.9 (520.8) -> 507.5 (518.0) MB, 8.0 / 0.0 ms  (average mu = 0.282, current mu = 0.256) allocation failure;
[6481:0x6e52790]     7195 ms: Scavenge (reduce) 508.2 (518.0) -> 507.6 (518.5) MB, 4.8 / 0.0 ms  (average mu = 0.282, current mu = 0.256) allocation failure;
[6481:0x6e52790]     7260 ms: Scavenge (reduce) 508.6 (518.5) -> 507.9 (518.5) MB, 1.6 / 0.0 ms  (average mu = 0.282, current mu = 0.256) allocation failure;


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb65a10 node::Abort() [node]
 2: 0xa761b5 node::FatalError(char const*, char const*) [node]
 3: 0xd5713e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xd574b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xf34545  [node]
 6: 0xf35448 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xf45953  [node]
 8: 0xf467c8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xf49995 v8::internal::Heap::HandleGCRequest() [node]
10: 0xec7aef v8::internal::StackGuard::HandleInterrupts() [node]
11: 0x12c64c5 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x16f87f9  [node]
Aborted (core dumped)

when i use yarn's resolutions to force google-gax 3.5.2 it works fine:

> node --max-old-space-size=512 test.js
[ [], { moreResults: 'MORE_RESULTS_AFTER_LIMIT', endCursor: 'CgA=' } ]

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[alexander-fenster]

Hi @scottbessler,

I cannot easily reproduce it (neither with an empty list of properties nor with non-empty list), so I will need to ask you to help us figure out what's going on.

Would you be able to try versions of google-gax between 3.5.2 and 3.5.8 and see which exact version starts failing? I believe there were no changes in google-gax that could change anything related to memory allocation, but there were two updates of dependencies (@grpc/grpc-js and protobufjs) and it would be very useful to see if we need to dig into the changes in either of them.

[weyert]

We have been having a memory leak to in one of the other google packages, probably, the @google-cloud/pubsub package and after downgrading to 3.5.7 the memory disappeared. Before we were using 3.6.0

[alexander-fenster]

@weyert Just to confirm - no memory leak on 3.5.7 but leaking on 3.5.8, with no other changes? The only difference between them is related to the HTTP transport https://github.com/googleapis/gax-nodejs/pull/1431/files which is very unlikely to be used by @google-cloud/pubsub.

Could you share more details, preferably a repro, in a new issue since it seems not directly related to what @scottbessler describes here?

[weyert]

@alexander-fenster I haven't tested 3.5.8. I will see what I can do to test the versions between 3.5.7 and 3.6.0. I can only say that there was an issue in 3.6.0 and force a downgrade to 3.5.7 resolved the issue by adding google-gax@3.5.7 as main dependency of the project.

I will report back as soon as possible. Now that a solution has been found for the memory leak I need to work through my backlog of tasks before I can do the above, though. I hope to be able to get back end of the week or early next week.

[leahecole]

going to close as obsolete given that nodejs-datastore now depends on 4.0.5 or greater, but please reopen if this is an issue still.