Profiling mmap on macOS not working

[herrmanno]

Problem

The heap profiler does not track mmap calls, even if environment variable HEAP_PROFILE_MMAP is set.

Environment

Key	Value
OS	macOS Monterey 12.6.9
gperftools:	36fa5ee9 (build from source)
Compiler	Apple clang version 14.0.0 (clang-1400.0.29.202)
Build-Tool	CMake 3.27.7

Minimal working example

foo.cpp

#include <sys/mman.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

int main() {

    int page_size = getpagesize();

    volatile size_t* a = (size_t*)mmap(0, 10 * page_size, PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE, -1, 0);
    //volatile size_t* a = (size_t*)malloc(10 * page_size);

    for (size_t i = 0; i < 100; i++) {
        a[i] = i;
    }

    volatile size_t n = 0;
    for (size_t i = 0; i < 100; i++) {
        n += a[i];
    }

    printf("OUT: %lu\n", n);
    
    return 0;
}

CMakeLists.txt

cmake_minimum_required(VERSION 3.27)

project(foo
    VERSION 0.1
    LANGUAGES CXX
)

find_library(TCMALLOC_LOCATION
    NAMES tcmalloc_and_profiler
    HINTS "${CMAKE_PREFIX_PATH}/"
)

add_library(tcmalloc SHARED IMPORTED GLOBAL)

set_target_properties(tcmalloc
    PROPERTIES
    IMPORTED_LOCATION ${TCMALLOC_LOCATION}
)

add_executable(
    foo
    foo.cpp
)

target_include_directories(foo PRIVATE ${CMAKE_SYSROOT}/usr/local/include/)
target_link_libraries(foo PRIVATE tcmalloc)

Steps to reproduce

Run ./build/foo binary as following:

HEAPPROFILE=foo HEAP_PROFILE_MMAP=true ./build/foo

Results

• Iff malloc was used for alloction the output says "Exiting, 44 kB in use", which seems correct.
• Iff mmap was used for allocation the output says "Exiting, 4 kB in use", which seems does not recognized the manually allocated memory. (4 kB seem to come from the printf call).

$ head foo.0001.heap (malloc)

heap profile:      2:    45056 [     2:    45056] @ heapprofile
     1:    40960 [     1:    40960] @ 0x7ff81a04eabb 0x104836eca 0x113e4852e
     1:     4096 [     1:     4096] @ 0x7ff81a04eabb 0x7ff81a0fa20b 0x7ff81a108dea 0x7ff81a0e2822 0x7ff81a0f5c56 0x7ff81a115069 0x104836f58 0x113e4852e

MAPPED_LIBRARIES:
7ff899f9e000-7ff8a2559844 r-xp 231a0000 00:00 0           /usr/lib/libc++.1.dylib
7ff85b93bed0-7ff85b93d0e8 r-xp 214a1ed0 00:00 0           /usr/lib/libc++.1.dylib
7ff85b992000-7ff85b99ae88 r-xp 214f8000 00:00 0           /usr/lib/libc++.1.dylib
7ff859f17af0-7ff859f1a2c8 r-xp 1fa7daf0 00:00 0           /usr/lib/libc++.1.dylib
7ff81a16a000-7ff81a1c3000 r-xp 00274000 00:00 0           /usr/lib/libc++.1.dylib

$ head foo.0001.heap (mmap)

heap profile:      1:     4096 [     1:     4096] @ heapprofile
     1:     4096 [     1:     4096] @ 0x7ff81a04eabb 0x7ff81a0fa20b 0x7ff81a108dea 0x7ff81a0e2822 0x7ff81a0f5c56 0x7ff81a115069 0x10e0b3f58 0x110ea052e

MAPPED_LIBRARIES:
7ff899f9e000-7ff8a2559844 r-xp 231a0000 00:00 0           /usr/lib/libc++.1.dylib
7ff85b93bed0-7ff85b93d0e8 r-xp 214a1ed0 00:00 0           /usr/lib/libc++.1.dylib
7ff85b992000-7ff85b99ae88 r-xp 214f8000 00:00 0           /usr/lib/libc++.1.dylib
7ff859f17af0-7ff859f1a2c8 r-xp 1fa7daf0 00:00 0           /usr/lib/libc++.1.dylib
7ff81a16a000-7ff81a1c3000 r-xp 00274000 00:00 0           /usr/lib/libc++.1.dylib
7ff899f9e000-7ff8a2559844 r-xp 231a0000 00:00 0           /usr/lib/liboah.dylib

Question

What exactly am I doing wrong? It seem my test program work correct. Linking seems to be correct, otherwise there should be no memory dump at all. The HEAP_PROFILE_MMAP seem to be correct according to the docs and true as enabling value seem to be correct according to source.

[alk]

Correct. We don't have working mmap interception on this platform. Feel free to contribute.

Here are the typical "sane" options:

a) define mmap/munmap and syscall directly bypassing libc's implementation. We do that on Linux and FreeBSD. Lately I saw OSX headers bark on syscall usage claiming it is deprecated. So maybe less of an option for OSX (or not, I'd check what they suggest).

b) define mmap/munmap and use dlsym with RTLD_NEXT to find OS's original mmap/munmap implementation. AFAIR previous FreeBSD mmap hooking implementation used that. Requires dynamic linking but perhaps that is already mandatory for your OS ?

c) sometimes libcs deliberately implement _mmap or something like that and mmap is just alias (the case for GNU libc). We then define our mmap with hooking functionality that eventually calls _mmap for actual mapping syscall.

On top of that there are some less sane options (like code patching and whatnot).

Good luck.

[herrmanno]

@alk Thanks for the clarification. I wasn't aware of that limitation on macOS (did I overlook some statements in the official documentation? ). If I find time I may look into your suggestions for contribution regarding the issue.

Right now I try using gperftools to measure memory usage for a scheduled project, so investigating the issue is no option (right now) for me. But just to be clear: If I was using Linux or (Free)BSD, my MWE would work as extected? In that case I might switch the platform for my current project.

Again, thanks for the quick and informative answer. That really helped me so far.

[alk]

FreeBSD and Linux should work, but note that we currently don't have mmap profiling test coverage. Another thing to keep in mind is issue #1128. That is both glibc and musl invoke internal "version" of mmap when mapping thread stacks, so we're unable to intercept those.