couchspawnkillable: Cannot allocate memory

[sergey-safarov]

Description

I use apache/couchdb:3.3.2 docker image on my server.
Today couchdb process crashed because no free memory on the server.
CouchDB logs contians

-- Logs begin at Tue 2024-01-09 16:32:03 UTC, end at Tue 2024-02-20 22:09:12 UTC. --
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: sh: 1: exec: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: sh: 1: exec: sh: 1: exec: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory/opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memorysh: 1: exec: sh: 1: exec: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: ^M
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: sh: 1: exec: sh: 1: exec: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: sh: 1: exec:
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:50:07 ippbx-0a.example.com docker[999209]: out of memory
Feb 20 21:06:23 ippbx-0a.example.com docker[999209]: out of memory
Feb 20 21:10:08 ippbx-0a.example.com docker[999209]: eheap_alloc: Cannot allocate 71302344 bytes of memory (of type "heap").
Feb 20 21:10:08 ippbx-0a.example.com docker[999209]: ^M^M
Feb 20 21:19:07 ippbx-0a.example.com docker[999209]: Crash dump is being written to: erl_crash.dump...
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: couchdb-docker.service: Main process exited, code=exited, status=137/n/a
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: couchdb-docker.service: Failed with result 'exit-code'.
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: couchdb-docker.service: Service RestartSec=100ms expired, scheduling restart.
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: couchdb-docker.service: Scheduled restart job, restart counter is at 2.
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: Stopped CouchDB container.

Steps to Reproduce

Do not know

Expected Behaviour

All external processes should not lead to all server memory usage.

Your Environment

• CouchDB version used: 3.3.2
• Browser name and version: do not used
• Operating system and version: CentOS 8 Stream, aarch64

Additional Context

Here is graph for swap memory usage on the server. Swap usage start grow about one week ago

And the disk cache started shrinking about 8 days ago

A related ticket found at https://issues.apache.org/jira/browse/COUCHDB-371 and looks like issue is not solved and closed with comment

If this issue remains in CouchDB 2.x, please file a new issue

[sergey-safarov]

Before the issue was used 170Mb of swap memory

[nickva]

I don't typically run CouchDB in a docker container so this all speculative advice.

A few things to try:

• Adjusting couchjs process setting, lower their number with https://docs.couchdb.org/en/stable/config/query-servers.html#query_server_config/os_process_limit. Make the hard limit say 75 and soft limit 25. The default is 100 I think.

• Try lowering max_dbs_open. The default is 500 so maybe try making it 400 or 300.

• When running in docker, Erlang VM doesn't always do a good job auto-detecting the CPU quotas so it would start a scheduler thread for each core on the host machine. So try lowering the number of schedulers artificially in vm.args with +S 4:4, for example to use only 4 schedulers. The number of schedulers should match your CPU quota.

• Erlang VM has more advanced memory management settings. There is one to use a max amount of heap, so if you have a way to test it out on a test cluster first, try something like:
  vm.args:

+MMscs 4096
+Musac false
+MMsco true

That sets the super-carrier to 4GB max. Again, I haven't tried running with those settings but someone reported some success with that: https://elixirforum.com/t/running-elixir-with-limited-heap-size/23633/13