OOM on 32GB AWS instance when hosting 5GB of data

[MrLukey]

Hi all. I am working on a scaled deployment of 3.1.2 Couchdb hosted on 60x AWS m7i.2xlarge (32GB RAM, 8 core) instances, and am struggling with very high memory use on all of them. Each scaled couch only supports 100 stores with a master (15MB), 2x cluster (12MB) and a customer (13MB) db, all of which contain ~900 documents. So in total each scaled instance is hosting ~5.2GB of data, and yet on all instances disk usage is over 22GB, and RAM used (after making updates to multiple docs without rebooting) is over 27GB, sometimes reaching the 32GB limit and falling over with OOM error (all of which consumed by beam.smp). Could someone please explain why this is, and how I can go about reducing the memory use in a significant way? I've tried bumping the Erlang version to 22.3.4.9, as it was pinned at 20.3.8.26 previously (and I had similar issues with Vernemq which were largely solved with a Erlang bump) but it hasn't made any difference. I also looked through various config setting but couldn't spot anything off with our current config. If you could point out what I've missed that would be a huge help. Thanks in advance.

ERL_FLAGS="-smp enabled +a 64 +Q 32768 +A 16 +SP 100:100"

[nickva]

Are there any crashes/exceptions in the log?

To get more details about the memory type beam is using could try GET-ing /_node/_local/_system or /_node/$nodename/_system on each of your nodes. That would return a result that might look something like:

{
    "context_switches": 118942,
    "distribution": {
           ...
        }
    },
    "ets_table_count": 199,
    "garbage_collection_count": 82297,
    "internal_replication_jobs": 0,
    "io_input": 5755,
    "io_output": 19117,
    "memory": {
        "atom": 631001,
        "atom_used": 599998,
        "binary": 121136,
        "code": 13297814,
        "ets": 1480752,
        "other": 27862305,
        "processes": 10754616,
        "processes_used": 10727208
    },
    "message_queues": {
        "alarm_handler": 0,
        "application_controller": 0,
       ...
    },
    "os_proc_count": 0,
    "process_count": 411,
    "process_limit": 262144,
    "reductions": 72481983,
    "run_queue": 0,
    "run_queue_dirty_cpu": 0,
    "stale_proc_count": 0,
    "uptime": 746,
    "words_reclaimed": 37034326
}

The memory object would have some more memory usage details.

Check your _active_tasks to see if there are compaction or index building jobs running. You could try lowering their concurrency perhaps https://docs.couchdb.org/en/stable/maintenance/compaction.html https://docs.couchdb.org/en/stable/config/indexbuilds.html

Some of flags ERL_FLAGS="-smp enabled +a 64 +Q 32768 +A 16 +SP 100:100" don't quite make sense for Erlang 22:

• +a and +A which configure the async thread pool, presumably for doing disk IO, are not effective. The async pool had been replaced with dirty IO schedulers. Those are configured via the +SDio flag. The latest CouchDB version uses 16 but https://github.com/apache/couchdb/blob/main/rel/overlay/etc/vm.args#L60-L62 on larger servers we've used 80 or 128 even.
• SMP should always be enabled and there is no need for the flag
• +SP 100:100 I think is the default anyway. It indicates using 100% of the detected CPU cores. In containerized environment the Erlang VM doesn't always properly read the configured cgroups CPU quota and might spawn a scheduler for each visible core on the host. Then it could make sense to manually lower the number of schedulers with say +S 8:8 or something.

Other flags I've seen successfully used in production on bare-metal nodes: -zdbbl32768 -Bd -SDio 80 -MBacul 0 -MBas aobf

Erlang 22 and 3.1.2 are quite a bit older. There have been some memory leak fixes in Erlang VMs since, if possible as a test try upgrading to the latest Erlang 24 patch version. CouchDB 3.1.x is also quite old and I think has a number of CVEs against it. See if it's possible to upgrade to 3.3.3.

I am not familiar with AWS's offering so not sure what m7i.2xlarge. Is that a containerized (Docker) environment? if that's the case then perhaps some of the advice here might help: #4988 (comment)

[MrLukey]

Thanks very much for you speedy response, you have given me plenty to think about:

• We cant upgrade to 3.3.3 yet due to a backwards incompatible api change to list functions, but are protected from CVEs - we will do this as soon as customers upgrade to a release with a compatible db
• the simulated stores take a while to reach the OOM state, so I will implement new memory metrics for next time
• thread pool / scheduler assignment / erlang memory assignment sounds most promising, as I have run into similar issues with containerised environments before - going to run some tests with various config changes suggested

Will let you know the results, and thanks again!

[MrLukey]

Ok, after a bit of testing I believe I have found the culprit, namely max_dbs_open, specifically the lingering processes associated with views created when modifying documents. On systems where max_dbs_open > no_of_dbs, there is never any reason to close the connections, so after modifying a number of documents, a huge number of view processes accumulate, which hog all the available memory.

dbs=404, max_dbs_open=1200:

root@zyncouchs1:~$ ls -l /proc/$(pidof beam.smp)/fd | grep \\\.couch | wc -l
804
root@zyncouchs1:~$ ls -l /proc/$(pidof beam.smp)/fd | grep \\\.view | wc -l
16391

When setting max_dbs_open < no_of_dbs then periodically connections to each db must be closed in order to open a new one (I assume there is a staleness attribute somewhere in the logic), which kills associated view processes and frees up any allocated memory. (we have implemented database sharding, so each store db requires 2 connections, hence 804 is the required value for all our dbs to have connections maintained permanently)

dbs=404, max_dbs_open=500:

root@zyncouchs3:~$ sudo ls -l /proc/$(pidof beam.smp)/fd | grep \\\.couch | wc -l
500
root@zyncouchs3:~$ sudo ls -l /proc/$(pidof beam.smp)/fd | grep \\\.view | wc -l
768

The impact on memory use is quite substantial, with only one additional change (setting number of schedulers available to Erlang with +S X:X where X is number of cores available) I was able to achieve an 80% reduction in memory use, all while running Erlang 20 and Couchdb 3.1.2 (both of which are very old).

I will certainly push to upgrade to the latest Erlang/Couchdb versions, and believe far more efficiency savings can be made there, but in the meantime I had hoped you could point me in the direction of a means to control the number of view processes allowed, without affecting the number of max_dbs_open, as for our purposes we want 804 permanent connections, but do not want to incur the cost of massive memory use!

Again thanks so much for your detailed and timely response, you have made me look like a bloody hero over here! ^_^

[nickva]

Good find about max_dbs_opens and thanks for sharing!

I had hoped you could point me in the direction of a means to control the number of view processes allowed, without affecting the number of max_dbs_open

I don't know if we have such an option, we had a limit in a much older version, but I don't think it's there any longer.

One more tweak that's available for db shards is how often should idle db shards stay around if they are idle. The current default is about 1 minute but that can be changed with (value is in milliseconds):

https://github.com/apache/couchdb/blob/main/src/couch/src/couch_db_updater.erl#L953

[couchdb]
idle_check_timeout = 61000

Maybe making it 15000 seconds could help.

[MrLukey]

Ok I'll test that out tomorrow / next week, but think the savings made already are enough for our purposes (we can cut the instance size to 1/4 of current MEM resource) - thanks again!