[macOS+macports+cmake] tcmalloc does not link on aarch64: Undefined symbols: _tc_delete_aligned etc.

[barracuda156]

Something was broken after 2.10 here:

[ 82%] Linking CXX shared library libtcmalloc.dylib
/opt/local/bin/cmake -E cmake_link_script CMakeFiles/tcmalloc.dir/link.txt --verbose=ON
/usr/bin/clang++ -pipe -Os -Wno-deprecated-declarations -Wno-error=unknown-warning-option -Wno-unknown-warning-option -DNDEBUG -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -fsized-deallocation -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -mmacosx-version-min=14.0 -dynamiclib -Wl,-headerpad_max_install_names -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -compatibility_version 9.14.5 -current_version 9.14.5 -o libtcmalloc.9.14.5.dylib -install_name @rpath/libtcmalloc.9.14.5.dylib CMakeFiles/tcmalloc.dir/src/tcmalloc.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/common.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/internal_logging.cc.o "CMakeFiles/tcmalloc_internal_object.dir/src/system-alloc.cc.o" CMakeFiles/tcmalloc_internal_object.dir/src/memfs_malloc.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/safe_strerror.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/central_freelist.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/page_heap.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/sampler.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/span.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/stack_trace_table.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/static_vars.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/symbolize.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/thread_cache.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/malloc_hook.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/malloc_extension.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/base/low_level_alloc.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/mmap_hook.cc.o "CMakeFiles/tcmalloc_internal_object.dir/src/heap-profile-table.cc.o" "CMakeFiles/tcmalloc_internal_object.dir/src/heap-profiler.cc.o" CMakeFiles/tcmalloc_internal_object.dir/src/raw_printer.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/emergency_malloc.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/emergency_malloc_for_stacktrace.cc.o CMakeFiles/tcmalloc_internal_object.dir/src/memory_region_map.cc.o CMakeFiles/stacktrace_object.dir/src/stacktrace.cc.o CMakeFiles/stacktrace_object.dir/src/base/elf_mem_image.cc.o CMakeFiles/stacktrace_object.dir/src/base/vdso_support.cc.o  -Wl,-rpath,/opt/local/lib libspinlock.a libsysinfo.a liblogging.a 
ld: Undefined symbols:
  _tc_delete_aligned, referenced from:
      operator delete(void*, std::align_val_t) in tcmalloc.cc.o
  _tc_delete_aligned_nothrow, referenced from:
      operator delete(void*, std::align_val_t, std::nothrow_t const&) in tcmalloc.cc.o
  _tc_delete_sized_aligned, referenced from:
      operator delete(void*, unsigned long, std::align_val_t) in tcmalloc.cc.o
  _tc_deletearray_aligned, referenced from:
      operator delete[](void*, std::align_val_t) in tcmalloc.cc.o
  _tc_deletearray_aligned_nothrow, referenced from:
      operator delete[](void*, std::align_val_t, std::nothrow_t const&) in tcmalloc.cc.o
  _tc_deletearray_sized_aligned, referenced from:
      operator delete[](void*, unsigned long, std::align_val_t) in tcmalloc.cc.o
  _tc_new_aligned, referenced from:
      operator new(unsigned long, std::align_val_t) in tcmalloc.cc.o
  _tc_new_aligned_nothrow, referenced from:
      operator new(unsigned long, std::align_val_t, std::nothrow_t const&) in tcmalloc.cc.o
  _tc_newarray_aligned, referenced from:
      operator new[](unsigned long, std::align_val_t) in tcmalloc.cc.o
  _tc_newarray_aligned_nothrow, referenced from:
      operator new[](unsigned long, std::align_val_t, std::nothrow_t const&) in tcmalloc.cc.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [libtcmalloc.9.14.5.dylib] Error 1

[barracuda156]

Identically fails with LLVM Clang 17.0.5.

[alk]

I need more details. You're using cmake? What is exact incantation you use ? (BTW you are aware that cmake support is deeply experimental and mostly 'use at your own risk'?) It compiles fine on my osx testing box (m1 macbook air).

[barracuda156]

@alk I will generate logs and share now. (With and without our patches.)

It is great that you got MBA M1: building via Macports should be reproducible and fast, if you have Sonoma installed. (Ventura will have pre-built ports, but I have no Ventura installation to check gperftools 2.13 build there at the moment.)

[barracuda156]

@alk So here are logs from trying to build 2.13 on Sonoma/aarch64 (same hardware with yours), with and without patches from my PR (but of those only ARM recognition is relevant, PowerPC stuff is inconsequential here, obviously):
gperftools_2.13_sonoma_xcode_clang_no_patches.txt
gperftools_2.13_sonoma_xcode_clang.txt

This commit updates the portfile: barracuda156/macports-ports-powerpc@171894f (it does not build for me atm on Sonoma, as described, so cannot be committed into Macports yet).

sudo port -v install gperftools should build the port; you could manually edit the portfile locally to build 2.13 instead of 2.10 which Macports currently has.

[alk]

I submitted your proposed changes. Thanks.

As for this specific ticket, I don't promise much attention. This seems specific to osx+cmake and perhaps plus whatever ports stuff you use. On my osx test box cmake nearly builds (there is one odd thing about weak functions that affects malloc_bench only; and somehow only happens with whatever flags or something cmake build does on osex). So your failures, I'd strongly prefer you investigate more yourself. Can we somehow separate those failures from ports stuff ?

Also please be aware of #1462

And finally, I don't approve that ports thingy choosing cmake build. We do have known issues there. And for example plenty of tests fail. If you can affect it somehow, please make it stop doing cmake stuff.

Or perhaps I should consider just amputating our "best but in practice not so good effort" cmake support?

[barracuda156]

@alk I will try building via auto tools and if that succeeds, try to figure out why CMake build fails.

[alk]

Ah btw if you intend to debug more of cmake stuff, here is malloc_bench patch that "fix" building with cmake on my osx test box:

diff --git a/benchmark/malloc_bench.cc b/benchmark/malloc_bench.cc
index 070e6e5..6f6bbdd 100644
--- a/benchmark/malloc_bench.cc
+++ b/benchmark/malloc_bench.cc
@@ -77,7 +77,7 @@ static void bench_fastpath_simple(long iterations,
   }
 }
 
-#ifdef __GNUC__
+#if defined(__GNUC__) && !defined(__APPLE__)
 #define HAVE_SIZED_FREE_OPTION
 
 extern "C" void tc_delete_sized(void *ptr, size_t size) __attribute__((weak));

I can simply merge it as is, but on the other hand, it would disable one feature even on autotools build where this at least compiles without trouble without patch. So perhaps we investigate some before doing more ifdefs mess.

[barracuda156]

@alk I will return to this in a couple of days, but as a quick update, the problem does not appear related to macOS version, arch or compiler. Here is the failure from 10.6 PowerPC with gcc13:

:info:build [ 71%] Linking CXX shared library libtcmalloc_minimal.dylib
:info:build /opt/local/bin/cmake -E cmake_link_script CMakeFiles/tcmalloc_minimal.dir/link.txt --verbose=ON
:info:build /opt/local/bin/g++-mp-13 -pipe -Os -DNDEBUG -I/opt/local/include -D_GLIBCXX_USE_CXX11_ABI=0 -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -fsized-deallocation -arch ppc -mmacosx-version-min=10.6 -dynamiclib -Wl,-headerpad_max_install_names -L/opt/local/lib -Wl,-headerpad_max_install_names -compatibility_version 9.14.5 -current_version 9.14.5 -o libtcmalloc_minimal.9.14.5.dylib -install_name @rpath/libtcmalloc_minimal.9.14.5.dylib CMakeFiles/tcmalloc_minimal.dir/src/tcmalloc.cc.o CMakeFiles/tcmalloc_minimal_internal_object.dir/src/common.cc.o CMakeFiles/tcmalloc_minimal_internal_object.dir/src/internal_logging.cc.o "CMakeFiles/tcmalloc_minimal_internal_object.dir/src/system-alloc.cc.o" CMakeFiles/tcmalloc_minimal_internal_object.dir/src/memfs_malloc.cc.o CMakeFiles/tcmalloc_minimal_internal_object.dir/src/safe_strerror.cc.o CMakeFiles/tcmalloc_minimal_internal_object.dir/src/central_freelist.cc.o CMakeFiles/tcmalloc_minimal_internal_object.dir/src/page_heap.cc.o CMakeFiles/tcmalloc_minimal_internal_object.dir/src/sampler.cc.o CMakeFiles/tcmalloc_minimal_internal_object.dir/src/span.cc.o CMakeFiles/tcmalloc_minimal_internal_object.dir/src/stack_trace_table.cc.o CMakeFiles/tcmalloc_minimal_internal_object.dir/src/static_vars.cc.o CMakeFiles/tcmalloc_minimal_internal_object.dir/src/symbolize.cc.o CMakeFiles/tcmalloc_minimal_internal_object.dir/src/thread_cache.cc.o CMakeFiles/tcmalloc_minimal_internal_object.dir/src/malloc_hook.cc.o CMakeFiles/tcmalloc_minimal_internal_object.dir/src/malloc_extension.cc.o  -Wl,-rpath,/opt/local/lib libspinlock.a libsysinfo.a liblogging.a 
:info:build Undefined symbols:
:info:build   "_tc_deletearray_aligned_nothrow", referenced from:
:info:build       __ZdaPvSt11align_val_tRKSt9nothrow_t in tcmalloc.cc.o
:info:build   "_tc_deletearray_sized_aligned", referenced from:
:info:build       __ZdaPvmSt11align_val_t in tcmalloc.cc.o
:info:build   "_tc_delete_sized_aligned", referenced from:
:info:build       __ZdlPvmSt11align_val_t in tcmalloc.cc.o
:info:build   "_tc_new_aligned_nothrow", referenced from:
:info:build       __ZnwmSt11align_val_tRKSt9nothrow_t in tcmalloc.cc.o
:info:build   "_tc_delete_aligned", referenced from:
:info:build       __ZdlPvSt11align_val_t in tcmalloc.cc.o
:info:build   "_tc_newarray_aligned", referenced from:
:info:build       __ZnamSt11align_val_t in tcmalloc.cc.o
:info:build   "_tc_new_aligned", referenced from:
:info:build       __ZnwmSt11align_val_t in tcmalloc.cc.o
:info:build   "_tc_newarray_aligned_nothrow", referenced from:
:info:build       __ZnamSt11align_val_tRKSt9nothrow_t in tcmalloc.cc.o
:info:build   "_tc_deletearray_aligned", referenced from:
:info:build       __ZdaPvSt11align_val_t in tcmalloc.cc.o
:info:build   "_tc_delete_aligned_nothrow", referenced from:
:info:build       __ZdlPvSt11align_val_tRKSt9nothrow_t in tcmalloc.cc.o
:info:build ld: symbol(s) not found
:info:build collect2: error: ld returned 1 exit status
:info:build make[2]: *** [libtcmalloc_minimal.9.14.5.dylib] Error 1

[barracuda156]

@alk Sorry for a delay, much stuff in the pipeline.

I can confirm that at least on macOS 14.2.1 gperftools 2.13 builds fine with makefiles and whatever options were picked by default (I did not customize the build, just commented out CMake-specific stuff).
gperftools_makefiles.log

[barracuda156]

While it still fails on the same OS with CMake:
gperftools_cmake_fail.log

[alk]

Thanks for update. Perhaps then consider running those same steps manually (check if some of your patches affect stuff; probably not, but who knows). I'd also compare more closely compiler flags.

[barracuda156]

@alk For sure patches have nothing to do with this. I reran the build, dropping all our patches, and it still failed with CMake the same way:
gperftools_cmake_default.log

[alk]

Consider checking what symbols are in debugallocation.cc and tcmalloc.cc where it fails to locate functions.

Something like this:

objdump -t src/.libs/libtcmalloc_minimal_la-tcmalloc.o

(No idea if osx has objdump but I think it should)

[barracuda156]

I'd also compare more closely compiler flags.

@alk Looks like CMake build adds -Dtcmalloc_minimal_debug_EXPORTS and -Dtcmalloc_EXPORTS which are not there in makefiles build. These look suspicious. Perhaps these are breaking it?

[alk]

No idea. If so why it only breaks it on your system ?

[barracuda156]

No idea. If so why it only breaks it on your system ?

Given that it is reproducible on two completely different systems (different OS versions, different archs, different compilers, different linkers, different standard library), I do not think it is anything related to a particular system. It could be that Macports CMake is doing something which causes the breakage or otherwise something in the source, or something with choice of options.

Let me try two things:

1. Build with CMake with no options from our side, so that defaults get used.
2. Try to pick options for CMake to match flags which get passed to the build with makefiles build.

[barracuda156]

@alk So I am sure it is something on the side of CMake build implementation. But it is partly fixed in the master (but broken in 2.13).

Specifically, with tests and benchmarks disabled the build is successful from the master branch 8987d08

With default setting build fails but on:

[ 54%] Linking CXX executable malloc_bench
/opt/local/bin/cmake -E cmake_link_script CMakeFiles/malloc_bench.dir/link.txt --verbose=ON
/usr/bin/clang++ -pipe -Os -Wno-deprecated-declarations -Wno-error=unknown-warning-option -Wno-unknown-warning-option -DNDEBUG -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -fsized-deallocation -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -mmacosx-version-min=14.0 -Wl,-search_paths_first -Wl,-headerpad_max_install_names -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk CMakeFiles/malloc_bench.dir/benchmark/malloc_bench.cc.o -o malloc_bench  -Wl,-rpath,/opt/local/lib librun_benchmark.a libtcmalloc_minimal.a libspinlock.a libsysinfo.a liblogging.a 
ld: Undefined symbols:
  _tc_delete_sized, referenced from:
      _main in malloc_bench.cc.o
      bench_fastpath_simple_sized(long, unsigned long) in malloc_bench.cc.o
  _tc_memalign, referenced from:
      _main in malloc_bench.cc.o
      bench_fastpath_memalign(long, unsigned long) in malloc_bench.cc.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [malloc_bench] Error 1

[alk]

Ah, I thought I merged workaround for this. Apple's stuff seemingly has borked weak symbols support.

Just replace ifdef __GNUC__ with something like defined(__GNUC__) && !defined(APPLE)

[barracuda156]

I can also confirm that without tests/benchmarks master branch builds on PowerPC as well with CMake.