Memory leak when running a Matterbridge with Wakuv2

[jakubgs]

Problem

This problem was discovered while working on deployment of our Matterbridge fork developed by @endulab 👍

• https://github.com/status-im/infra-misc/issues/250

Despite increasing host size the memory usage keep growing until the host is full and SWAPping starts:

This issue is unlikely to be in the Matterbridge code as it's a well established software, and our Waku additions are minimal.

[jakubgs]

Granted Michał SSH access to use pperf over SSH:

• infra-misc#87cf8427 - bridge: add nologin SSH access for michal@status.im
• feat: enabling pprof to receive memory reports matterbridge#8

[endulab]

Created pprof reports.
Already sent to @richard-ramos report with memory used.

[jrainville]

Could be fixed by #5012

[endulab]

I updated bridge with the latest status-go. Let's see what happens. @jakubgs please update the bridge.

status-im/matterbridge#8

[jakubgs]

The port 6060 was not mapped from the container:

jakubgs@node-01.do-ams3.bridge.misc:/docker/matterbridge-community % grep -A5 ports docker-compose.yml
    ports:
      - '4343:4343'

Fixed it manually for now:

jakubgs@node-01.do-ams3.bridge.misc:/docker/matterbridge-community % sudo netstat -lpnt | grep 6060
tcp        0      0 0.0.0.0:6060            0.0.0.0:*               LISTEN      911/dockerd   

It works over SSH tunnel: ssh -N -L 6060:0:6060 node-01.do-ams3.bridge.misc.status.im

 > curl -s 0:6060/debug/pprof/cmdline | cat 
/bin/matterbridge-conf=/config/matterbridge.yaml

[jrainville]

release/0.177.x now contains #5061
Can we update the bridge to point to 79055f10aac2a0ed97fe022329afcab52dcfcfc6 or the HEAD of release/0.177.x? It should fix the issue.

[endulab]

@jakubgs I upgraded the bridge. It contains memory issue fix.
Please use again this branch and lets monitor.
FYI @jrainville

[jakubgs]

Thanks. I've built the updated enabling-pprof branch and pushed the image:

jakubgs@node-01.do-ams3.bridge.misc:/docker/matterbridge-community % d
CONTAINER ID   NAMES                    IMAGE                                       CREATED          STATUS
d294d6e741df   matterbridge-community   statusteam/matterbridge:1.26.1-dev-wakuv2   10 seconds ago   Up 8 seconds
fe68655db556   matterbridge-legacy      statusteam/matterbridge:v1.26.0-status      6 weeks ago      Up 10 days

Lets see what happens.

[endulab]

Last 3 days. Memory consumption on a constant level:

I will upgrade wakuv2 branch with a new version of status-go and turn off the profiler.