[Core] Possible memory leak in gcs_server

[ScottShingler]

What happened + What you expected to happen

Running Ray Core using a driver that repeatedly invokes a method on an actor results in a continuous overall increase in the peak RSS memory usage of the ray/core/src/ray/gcs/gcs_server process:

Over the course of ~112 hours, the values of the peaks increased from an initial value of ~595 MB to ~662 MB. That works out to roughly 598 KB per hour.

This plot and the data used to create it can be downloaded here.

Versions / Dependencies

ray 2.20.0

Reproduction script

I have created a repo that contains the reproduction script, as well as scripts to collect and plot the data: https://github.com/Prolucid/ray-memleak-repro.

Issue Severity

Medium: It is a significant difficulty but I can work around it.