Understanding Linking and Stripping

[olokobayusuf]

Hey all, I'm trying to wrap my head around how the Emscripten toolchain handles stripping specifically around JS library symbols. My use case is simple: I'm compiling a main module that will be extended at runtime with dynamic linking (side module). The main module needs to contain embind symbols that the side module will try to use, but since these symbols are not used at compile-time, they will get stripped away. Here's what I've learned so far:

1. Unused symbols defined in JS library functions (e.g. LibraryEmVal) will be stripped as expected.
2. wasm-ld does not have visibility or control over JS library symbol stripping, so using -Wl,-u,unused_symbol_name does not work.
3. Similarly, wasm-ld cannot preserve extern symbols (i.e. symbols defined in JS), so using -Wl,--export=unused_symbol_name also does not work.

My question then is what part of the Emscripten toolchain handles stripping JS library functions, and what compiler flags can be used to preserve these symbols?

I've tested -sEXPORTED_FUNCTIONS which doesn't work because this controls making C symbols visible in JavaScript, not the other way around. Similarly, -sEXPORTED_RUNTIME_METHODS doesn't work because it only handles making a function available in the global Module, and doesn't preserve any form of stripping.

[sbc100]

If you build with -sMAIN_MODULE then emscripten won't strip anything and it will include basically everything by default.

If you build with -sMAIN_MODULE=2 then emscripten will strip dead code. The easiest way to mark things are needed is to simply pass the side module on the command line when linking the main module. Then any symbols needed by the side module will be automatically preserved. If you don't have access to the side module(s) when linking the main module and you still want to use -sMAIN_MODULE=2 then you need to add all symbols needs by the side modules(s) explicitly using -sEXPORTED_FUNCTIONS.

There are some docs on this here: https://emscripten.org/docs/compiling/Dynamic-Linking.html#code-size

[olokobayusuf]

Integrated this and ran some more tests. I've pinpointed why my code still doesn't work, hopefully you can shed some light:

1. First, I'm currently using Emscripten 3.1.8 but the version doesn't quite matter for the mechanics.
2. Your suggestion on using -sDEFAULT_LIBRARY_FUNCS_TO_INCLUDE is spot on; the necessary JS library function from embind is now available in the module.
3. My side module gets loaded at runtime using dynamic linking, and needs to use aforementioned JS library function from the DCE'd main module. From the dynamic linking implementation, the WebAssembly.Module is created with a set of imports from the main module (providing things like cstdlib and other runtime methods). The issue is that there doesn't seem to be a way to add a symbol into asmLibraryArg unless it is explicitly used by the main module.

In 3.1.8, asmLibraryArg is emitted by create_module:

emscripten/emscripten.py

Lines 837 to 842 in 3ff7eff

	def create_module(sending, receiving, invoke_funcs, metadata):
	invoke_wrappers = create_invoke_wrappers(invoke_funcs)
	receiving += create_named_globals(metadata)
	module = []
	module.append('var asmLibraryArg = %s;\n' % sending)

And sending is in turn created from the compiled WASM binary metadata (I presume this comes from the output of wasm-ld without any modifications):

emscripten/emscripten.py

Lines 719 to 724 in 3ff7eff

	def create_sending(invoke_funcs, metadata):
	em_js_funcs = set(metadata['emJsFuncs'].keys())
	declares = [asmjs_mangle(d) for d in metadata['declares']]
	externs = [asmjs_mangle(e) for e in metadata['globalImports']]
	send_items = set(invoke_funcs + declares + externs)
	send_items.update(em_js_funcs)

And metadata extraction is implemented here:

emscripten/tools/extract_metadata.py

Lines 165 to 176 in 3ff7eff

	def extract_metadata(filename):
	module = webassembly.Module(filename)
	export_names = []
	declares = []
	invoke_funcs = []
	imported_funcs = 0
	imported_globals = 0
	global_imports = []
	em_js_funcs = {}
	exports = module.get_exports()
	imports = module.get_imports()
	globls = module.get_globals()

Both metadata["declares"] and metadata["globalImports"] are populated from webassembly.Module.get_imports(), so my question is what's missing? How come if I add the following line of code, _emval_get_global is properly declared as an import in my main module:

auto someGlobal = emscripten::val::global("window");

But using combinations of sDEFAULT_LIBRARY_FUNCS_TO_INCLUDE, -sEXPORTED_FUNCTIONS, -Wl,-u_emval_get_global, and -Wl,--export-if-defined=__emval_get_global all don't work?

[sbc100]

The symbols available to the side module (at least back in 3.18) are not only the ones in asmLibraryArg, but also any symbols exported on the Module object. See resolveGlobalSymbol.

However, a lot of this code as been updated in recent versions of emscripten. Is there any reason why you can't update to a more recent version. 3.1.8 is fairly old at this point.

Also, I'm not sure you answer the question: "Are you able you pass the side modules are augments when linking the main module?".

[olokobayusuf]

The symbols available to the side module (at least back in 3.18) are not only the ones in asmLibraryArg, but also any symbols exported on the Module object. See resolveGlobalSymbol.

Good catch! I'm getting very close to resolving this. resolveGlobalSymbol has fallbacks to search for the missing symbol, in this case symName is _emval_get_global as an example:

emscripten/src/library_dylink.js

Lines 10 to 30 in 3ff7eff

	$resolveGlobalSymbol: function(symName, direct) {
	var sym;
	#if !WASM_BIGINT
	if (direct) {
	// First look for the orig$ symbol which is the symbols without
	// any legalization performed.
	sym = asmLibraryArg['orig$' + symName];
	}
	#endif
	if (!sym) {
	sym = asmLibraryArg[symName];
	}
	// Check for the symbol on the Module object. This is the only
	// way to dynamically access JS library symbols that were not
	// referenced by the main module (and therefore not part of the
	// initial set of symbols included in asmLibraryArg when it
	// was declared.
	if (!sym) {
	sym = Module[asmjsMangle(symName)];
	}

asmjsMangle(symName) in this case adds a leading underscore, so we search for Module["__emval_get_global"]. The problem though that for whatever reason, I can't get the __emval_get_global JS function (which exists thanks to -sDEFAULT_LIBRARY_FUNCS_TO_INCLUDE) to get exported into Module with -sEXPORTED_RUNTIME_METHODS. Using -sEXPORTED_FUNCTIONS=__emval_get_global doesn't help.

An interesting thing I noticed is this declaration in the resulting JS framework code:

unexportedRuntimeFunction('emval_get_global', false);

The embind JS library defines both emval_get_global and __emval_get_global, with the latter depending on the former. I can get emval_get_global added to Module using -sEXPORTED_RUNTIME_METHODS, but the same doesn't apply for __emval_get_global.

[olokobayusuf]

However, a lot of this code as been updated in recent versions of emscripten. Is there any reason why you can't update to a more recent version. 3.1.8 is fairly old at this point.

We're currently adding support for Unity 2022, which uses Emscripten 3.1.8. Later versions of Unity have bumped their Emscripten version, but we need to support Unity versions all the way back to 2022.

Also, I'm not sure you answer the question: "Are you able you pass the side modules are augments when linking the main module?".

Oh sorry. We can't provide side modules when compiling the main module. We have a pretty advanced use case: we're building a cross-platform Python function runtime, where we transpile and cross-compile Python functions then ship these compiled binaries to users' devices at runtime. To power all this, we exploit a ton of dynamic linking across Android, iOS, macOS, Windows, and WebAssembly. Check us out.

[sbc100]

I assume this is because you don't know that total set of possible side modules that you might load? In which case, one shortcut to getting all the symbols you need might be to build a "sample"/"superset" side module that does a superset of things that you want to enable (.e.g it uses embind) and then link against that?

[sbc100]

Should I open a PR for this?

Sure that sounds reasonable.

[sbc100]

asmjsMangle(symName) in this case adds a leading underscore, so we search for Module["__emval_get_global"]. The problem though that for whatever reason, I can't get the __emval_get_global JS function (which exists thanks to -sDEFAULT_LIBRARY_FUNCS_TO_INCLUDE) to get exported into Module with -sEXPORTED_RUNTIME_METHODS. Using -sEXPORTED_FUNCTIONS=__emval_get_global doesn't help.

When I try this on the (latest emscripten version at least) it what you want:

$ emcc test/hello_world.c -lembind -sMAIN_MODULE=2 -sDEFAULT_LIBRARY_FUNCS_TO_INCLUDE=_emval_get_global -sEXPORTED_FUNCTIONS=__emval_get_global,_main
$ grep __emval_get_global a.out.js
...
  Module['__emval_get_global'] = __emval_get_global;
...

Does this not work for you? It sounds like add __emval_get_global to EXPORTED_FUNCTIONS does not work.. but if a you try to add a symbol that desn't exist to that list then you would get an error. e.g.

$ ./emcc ~/test/hello.c -lembind -sMAIN_MODULE=2 -sDEFAULT_LIBRARY_FUNCS_TO_INCLUDE=_emval_get_global -sEXPORTED_FUNCTIONS=missing,_main
emcc: error: undefined exported symbol: "missing" [-Wundefined] [-Werror]

[sbc100]

It also seems to work for me with 3.1.8:

$ emcc ~/test/hello.c -lembind -sMAIN_MODULE=2 -sDEFAULT_LIBRARY_FUNCS_TO_INCLUDE=_emval_get_global -sEXPORTED_FUNCTIONS=__emval_get_global,_main
$  grep __emval_get_global a.out.js
  function __emval_get_global(name) {
  Module["__emval_get_global"] = __emval_get_global;
  __emval_get_global.sig = 'ii';

[olokobayusuf]

No clue here, but I decided to include the symbols through source code (write nop code that uses the required symbols) instead of trying to do this through compiler and linker flags.

Now, I have a separate issue that is unrelated but since I have your attention I thought to go ahead and ask asnyway (lmk if I should create a separate discussion, thanks):

I am trying to use emscripten::val without RTTI (and the requiredEMSCRIPTEN_HAS_UNBOUND_TYPE_NAMES 0 definition) but at runtime I get an unbound type: emsc error. You've previously worked on this (here), and fixed it (here) but I'm struggling to understand how your fix actually works.

I think I'm facing this bug because of my custom setup: I have a main module being compiled with -lembind -fno-rtti, and I'm trying to use emscripten::val::as<T> in a side module that gets dynamically linked into the main module at runtime. I believe that based on the current implementation of emscripten::val (and the underlying wire impl), it uses a sentinel value to tell apart types at build time when -fno-rtti is used (i.e. in place of using typeid when using RTTI). But with a dynamically-linked side module, the type indices don't exist, so I get an invalid placeholder type emsc. Is my intuition correct?

If my intuition is correct, I can imagine a workaround where I patch requireRegisteredType JS library implementation (embind.js) to find the correct type. The issue though is that I can't do a type name search through registeredType because getTypeName doesn't seem to work when EMSCRIPTEN_HAS_UNBOUND_TYPE_NAMES == 0:

emscripten/system/lib/embind/bind.cpp

Lines 45 to 49 in 9eff02b

	} else {
	char str[80];
	sprintf(str, "%p", reinterpret_cast<const void*>(ti));
	return strdup(str);
	}

What do you think?

[olokobayusuf]

This is the workaround I'm referring to, which would only work with RTTI.