Default implementation of MallocExtension::GetAllocatedSize when initializing a static variable in a shared library.

[poljak181]

Hi!
I have a problem similar to this issue #473.
But I don't understand how to solve it.

Here is an example.
Code of the shared library where a static variable is initialized.

#include <iostream>
#include <stdlib.h>

static int* own_new_int_value = nullptr;
static bool Init() {
    std::cerr << "call static Init() from libown_new.so\n";
    if (!own_new_int_value) {
	own_new_int_value = new int;
	*own_new_int_value = 100500;
    }

    return true;
}
static bool inited = Init();

void lib_own_new_print()
{
    std::cerr << "lib_own_new_print start\n";
    {
	auto pi = new int;
	*pi = 42;
	delete pi;
    }
    std::cerr << "lib_own_new_print end\n";
}

Command to compile it:

clang++ -g -shared -fPIC -o libown_new.so lib_own_new.cpp

Main program

#include <iostream>
#include <stdlib.h>
#include <gperftools/tcmalloc.h>
#include <gperftools/malloc_extension.h>

void* operator new(size_t size)
{
    std::cerr << "	main.cpp: operator new\n";
    auto ptr = tc_malloc(size);
    std::cerr << "	allocated:size:" << MallocExtension::instance()->GetAllocatedSize(ptr) << "\n";
    return ptr;
}

void operator delete(void* p) _GLIBCXX_USE_NOEXCEPT
{
    std::cerr << "	main.cpp: operator delete\n";
    tc_free(p);
}

void lib_own_new_print();

int main()
{
    std::cerr << "start main\n";
    {
	auto pi = new int;
	*pi = 42;
	delete pi;
    }
    lib_own_new_print();
    std::cerr << "end main\n";
    return 0;
}

Command to compile the main program

clang++ -g main.cpp -ltcmalloc_minimal -L. -lown_new

Start the program with

LD_LIBRARY_PATH=. ./a.out

The result is:

call static Init() from libown_new.so
        main.cpp: operator new
        allocated:size:0
start main
        main.cpp: operator new
        allocated:size:8
        main.cpp: operator delete
lib_own_new_print start
        main.cpp: operator new
        allocated:size:8
        main.cpp: operator delete
lib_own_new_print end
end main

As you can see, when the inited variable from libown_new is initialized with the Init() function,

MallocExtension::Register(new TCMallocImplementation);

hasn't been called yet. So, the call to

MallocExtension::instance()->GetAllocatedSize(ptr);

in my operator new returns 0.

Full example attached
tcmalloc_extension.zip

[alk]

thanks for filing the ticket. It is actually fixable. Our current behavior is artifact of Google's internal situation where malloc extensions are useful even in build configurations that don't do tcmalloc (e.g. when asan's malloc is used). I am currently making significant change in hooks and initialization bits, so will try to get this done too. Expect something available to test and review (every pair of eyes is helpful) in next few days.

[alk]

And you should also be able to play "games" with linking order. If you link libtcmalloc or libtcmalloc_minimal sooner than other .so modules, it should be initialized before them.

[poljak181]

Thanks for the response!
Good news that upcoming changes may help with this problem!

If I understand correctly, there is no good way to control the order of dynamic library loading in Linux. Additionally, I tried to examine the current loading order using the command

LD_DEBUG=files ./a.out

It seems that tcmalloc is loaded first

     15375:     file=libtcmalloc_minimal.so.4 [0];  needed by ./a.out [0]
     15375:     file=libtcmalloc_minimal.so.4 [0];  generating link map
     15375:       dynamic: 0x00007fb5ce366bb0  base: 0x00007fb5ce33f000   size: 0x00000000001daee0
     15375:         entry: 0x00007fb5ce33f000  phdr: 0x00007fb5ce33f040  phnum:                 12
     15375:
     15375:
     15375:     file=libown_new.so [0];  needed by ./a.out [0]
     15375:     file=libown_new.so [0];  generating link map
     15375:       dynamic: 0x00007fb5ce33dde0  base: 0x00007fb5ce33a000   size: 0x0000000000004088
     15375:         entry: 0x00007fb5ce33a000  phdr: 0x00007fb5ce33a040  phnum:                  9
     15375:
     15375:
     15375:     file=libstdc++.so.6 [0];  needed by ./a.out [0]
     15375:     file=libstdc++.so.6 [0];  generating link map
     15375:       dynamic: 0x00007fb5ce332c90  base: 0x00007fb5ce10e000   size: 0x000000000022b8c0
     15375:         entry: 0x00007fb5ce10e000  phdr: 0x00007fb5ce10e040  phnum:                 12

[alk]

I think what happens is libtcmalloc{,_minimal) depends on libstdc++ which .init bits call malloc. So we end up calling our internal malloc initialization bits (because we have to perform that malloc request) before any of our own initialization runs.

As for the order of loading, I am pretty sure that this is well defined (breath first of something; google around).

[poljak181]

I used another approach to observe the initialization order of libraries' static variables:

 LD_DEBUG=files ./a.out 2>&1 | grep "calling init"

The result is

      3774:     calling init: /lib64/ld-linux-x86-64.so.2
      3774:     calling init: /lib/x86_64-linux-gnu/libc.so.6
      3774:     calling init: /lib/x86_64-linux-gnu/libgcc_s.so.1
      3774:     calling init: /lib/x86_64-linux-gnu/libm.so.6
      3774:     calling init: /lib/x86_64-linux-gnu/libstdc++.so.6
      3774:     calling init: ./libown_new.so
      3774:     calling init: /lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4

It turns out that fixing the linking order resolves the issue! I moved -ltcmalloc_minimal to the end:

clang++ -g main.cpp -L. -lown_new  -ltcmalloc_minimal

Now, the result of

LD_DEBUG=files ./a.out 2>&1 | grep "calling init"

is

      3765:     calling init: /lib64/ld-linux-x86-64.so.2
      3765:     calling init: /lib/x86_64-linux-gnu/libc.so.6
      3765:     calling init: /lib/x86_64-linux-gnu/libgcc_s.so.1
      3765:     calling init: /lib/x86_64-linux-gnu/libm.so.6
      3765:     calling init: /lib/x86_64-linux-gnu/libstdc++.so.6
      3765:     calling init: /lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4
      3765:     calling init: ./libown_new.so

But unfortunately, I still haven't found any official documentation explicitly stating that the loader will initialize the static variables of dynamic libraries in the reverse order specified during linking.