Is there any way to calculate or predict the total memory usage of cachebench?

[hyeongtakji]

Hello,

I am currently using https://github.com/facebook/CacheLib/blob/v2023.07.17.00/cachelib/cachebench/test_configs/hit_ratio/cdn/config.json
this config file. When I check the total memory usage of cachebench, I roughly get about 11GB.

$ ./bin/cachebench --json_test_config ./test_configs/hit_ratio/cdn/config.json
$ numastat -c -p 3746399

Per-node process memory usage (in MBs) for PID 3746399 (cachebench)
         Node 0 Node 1 Node 2 Node 3 Total
         ------ ------ ------ ------ -----
Huge          0      0      0      0     0
Heap         99     38    111     35   282
Stack         0      0      0      0     0
Private    3313   1789   4074   1844 11020
-------  ------ ------ ------ ------ -----
Total      3412   1827   4185   1879 11303

When I modify config.json a bit, the memory usage result changes as well (to about 19GB).

$ diff ./test_configs/hit_ratio/cdn/config.json ./test_configs/hit_ratio/cdn/config_new.json
23c23
<       "numOps": 5000000,
---
>       "numOps": 50000000,
$ ./bin/cachebench --json_test_config ./test_configs/hit_ratio/cdn/config_new.json
$ numastat -c -p cachebench

Per-node process memory usage (in MBs) for PID 3744509 (cachebench)
         Node 0 Node 1 Node 2 Node 3 Total
         ------ ------ ------ ------ -----
Huge          0      0      0      0     0
Heap         98     36    104     44   282
Stack         0      0      0      0     0
Private    6730   2775   6754   3032 19291
-------  ------ ------ ------ ------ -----
Total      6828   2811   6858   3076 19573

Modifying numThreads also makes some differences.

So my question is:

1. Can anyone explain how numOps or numThreads affects the total memory usage of cachebench?
2. Or if possible, it would be great if there is any way to calculate the total memory usage just by reading config file before running the benchmark.

[haowu14]

In general the memory consumption of cachebench has three parts: 1) the space that's used for caching (the slab space); 2) the memory consumption of cachelib operation (index, NVM buffer, etc); 3) cachebench operational usage.

1. is configured to be 8GB from the json file. (mmap)
2. Without NVM cache, this is usually negligible. With NVM cache, this is usually dominated by bloom filter for BigHash, NVm buffer and index for BlockCache.
3. Each of cachebench thread would create a bit of overhead since they do things like generating key, sampling. But they are usually pretty small as well. This could be affected by numThreads.

in general numOps should'nt affect the memory consumption by too much.

In your case, can you walk me through what each row of the output is?

[hyeongtakji]

FYI)
While modifying numOps (from 5m to 50m) increased the memory usage about 8GB (from 11GB to 19GB), modifying numThreads (from 48 to 12) decreased the memory usage only about 1GB (from 11GB to 10GB).

[haowu14]

Oh, I meant what's the difference in Huge, Heap, Stack, Private in numastack.

[hyeongtakji]

I used numastat just to check the total memory usage of a process. AFAIK, it parses /proc/pid/numa_maps and show the memory usage of each process per NUMA node and per category (heap, stack, etc). Like:

$ cat /proc/self/numa_maps
...
55a73ca61000 default heap anon=3 dirty=3 active=0 N1=3 kernelpagesize_kB=4 (3KB in node1 for heap)
...

Sorry for confusing information. Simple ps would have been better for this time.

$ ./bin/cachebench --json_test_config ./test_configs/hit_ratio/cdn/config_new.json (numOps = 50m)
$ ps ux | grep cachebench
hyeongt+ 4136101 2729  4.3 23556868 20026132 pts/57 Sl+ 11:40  29:07 ./bin/cachebench --json_test_config ./test_configs/hit_ratio/cdn/config_new.json

We can see when numOps is 50m, RSS of cachebench is about 19 GB (20026132 KB)

[haowu14]

I see. I am able to reproduce a similar memory consumption.
The key to your question is that:

1. cachebench generates all the operations before starting the actual simulation.
2. the total number of operations is numOps*numThreads. (For example 50m * 48 = 2400m).

Outside of the slab space, the most expensive thing is here: https://github.com/facebook/CacheLib/blob/main/cachelib/cachebench/workload/WorkloadGenerator.cpp#L174C19-L174C19
In the case where there are 2400m operations, that's 2400m * sizeof(uint32_t) ~ 9GB.
The other memory consuming things are:

1. Keys: there will be 8935378 keys (with some adjustments, but this is the order of magnitude around 9M). But each key's length is drawn from a distribution. So It's hard to estimate this. I added a calculation into the code and ran this test config. On one of my experiment that is about 200MB. https://github.com/facebook/CacheLib/blob/main/cachelib/cachebench/workload/WorkloadGenerator.cpp#L78
2. Reqs: The request and sizes (if it's a range) of the request. In my experiment that's 800MB. https://github.com/facebook/CacheLib/blob/main/cachelib/cachebench/workload/WorkloadGenerator.cpp#L130

So we have around 1GB here. With 9GB in the operations then 8GB in the slab space, adding up to 18GB, close to the 19GB number you have.

I think the reduction of memory when you reduce numOps or numThreads can be explained mostly by the reduction of the total number of operations.

[hyeongtakji]

The explanation about total number of operations is clear and helped me a lot. Really appreciate it.
It seems like now I can roughly predict the memory consumption before running cachebench.