Uploading files to GCS via WriteChannel cause the memory to increase - memory not released

[ihudedi]

Hi
I am uploading files to GCS bucket using WriteChannel
WriteChannel writeChannel = storage.writer(blobInfo);
writeChannel.write(byteBuffer);
writeChannel.close();

After many files being uploaded the memory increased and doesn't released.

Thanks,
Itay

[BenWhitehead]

Hi @ihudedi,

When creating a WriteChannel an internal buffer is created to perform chunking of the upload to GCS.

By default, the buffer size is 16MiB. Depending on which JVM you're using along with which garbage collector, the allocated buffers can sometimes be immediately placed into the Old Generation of the heap. Once in the Old Generation, they only way they are garbage collected is when a full collection cycle happens. If you're app is not generating enough garbage to cause the Old Generation to need a full collection often the JVM won't.

If the objects you are uploading are smaller than 16MiB, it probably makes sense to reduce that size by doing

try (WriteChannel writeChannel = storage.writer(info)) {
  writeChannel.setChunkSize(4 * 1024 * 1024); // 4MiB instead of the default of 16MiB
  writeChannel.write(byteBuffer);
}

If you're able, could you share which JVM, version and GC settings you're using?

[ihudedi]

Hi @BenWhitehead
java version "11.0.10" 2021-01-19 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.10+8-LTS-162)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.10+8-LTS-162, mixed mode)
-XX:+UseParallelGC -XX:-TieredCompilation -Xmx2048m -Xshare:off -XX:MaxDirectMemorySize=512m -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=10 -XX:+HeapDumpOnOutOfMemoryError