Tracking issue for wasm support

[brson]

It's been a while since I've looked at the state of wasm in Rust, and
there's a lot more to be done for it to be really usable. Is anybody
doing anything significant with Rust and emscripten yet?

Today I've triaged all the emscripten issues and put together a little
roadmap. I'm hoping others can help fill in the gaps here and get this
across the finish line.

I'll start by outlining the near-term scope, some discussion of the
state of things, then the tasks.

Goal: Make Rust on wasm + emscripten a reliable, 1st class Rust target.

Success metrics:

• Cross-compilation works from macOS, Windows and Linux
• Tests are running against wasm on the bots
• All common crates without significant platform or I/O deps work as expected
  and pass their test suites
• One can test wasm via the 'cross' tool or some other tool
• The rustc and emcc LLVM versions are decoupled
• One can tweak relevant emcc options in a consistent way
• Rust on emcc is as fast as possible when building in release mode

Out of scope:

• Automatic installation of emscripten
• Retaining asm.js compatibility

Discussion

Today I read reports of problems compiling from Windows and macOS;
some important crates that should obviously work just don't, like
regex; I have no idea whether the test suite still passes.

Setting up emcc is not foolproof, but I don't think we should worry
about solving it now, since no other cross target solves the problem
of acquiring the linker. Perhaps something like the cross tool will
suffice - it or something like it will be needed for testing emcc
targets anyway.

The issue of LLVM coupling remains the worst, and with the pending
llvm upgrade it's a pressing matter. To recap, today rustc uses the
emcc 'fastcomp' LLVM backend to communicate LLVM IR to emcc, to
compile to wasm/asm.js. This forces rustc and emcc to share an LLVM
fork, which is not sustainable (emcc LLVM development is basically
complete - the only reason for them to upgrade LLVM is because we
upgrade LLVM).

The only viable solution I see is to migrate to the upstream LLVM wasm
backend, changing the rustc wasm model such that rustc produces wasm
directly, and emcc links that. And this model unfortunately
precludes asm.js support until somebody writes a wasm->asm.js
translator. I think this is just the way it has to be. The future is
wasm, and we can't maintain LLVM in sync with emcc for long.

I don't think though we can do the migration to the LLVM wasm backend
before the next LLVM upgrade though, because that backend is not
ready. So we're probably going to be asking @azakai to upgrade emcc to
LLVM 4.0 and continuing with the LLVM IR -> emcc -> wasm path for a
while.

Tasks

• Upgrade LLVM to 4.0
• macOS link failures
• emscripten support broken by cargo file layout changes
• binaryen fails to validate wasm
• remove exception handling code with panic runtime
• optimize emscripten targets with emcc
• llc crashes linking regex test suite
• Get test suite working with wasm
• Set up testing bots for wasm and asm.js
• Run all crates.io tests against wasm to discover bugs
• Tool support for testing wasm targets
• Migrate wasm target to LLVM wasm backend
• Configure C exports correctly
• mistranslated wasm simd instructions
• make sure asm.js/wasm extensions are correct

[kripken]

Out of scope: [..] Retaining asm.js compatibility

Just curious, why? Doing asm.js first, which is very stable, is the more reliable path, instead of going straight to wasm. It's true it requires the fastcomp update, which might or might not be a lot of work, which I don't think we know yet.

[est31]

@kripken have you read further below? It seems that @brson wants to use LLVM's wasm backend, which would apparently mean that asm.js compat is dropped. It would make sense as then emcc wouldn't block the LLVM upgrade any more.

[kripken]

Yeah, that's a bunch of additional questions, though, as currently it is not yet stable and produces larger/slower code. But it does have the advantage of not maintaining an LLVM fork, which is considerable of course.

In principle, if we want to do the least work for rust-on-the-web, we could wait for the wasm backend or use it with the downsides, if we don't mind them.

[chicoxyzzy]

I'm not sure if this is a good place for my comment but I don't know where to create an issue. Please point me to the right repo in case it's off topic.

I was following instructions from https://users.rust-lang.org/t/compiling-to-the-web-with-rust-and-emscripten/7627 and had no luck in generating working example.

• Resulting JS file doesn't work on node. Error is

no binaryen method succeeded. consider enabling more options, like interpreting, if you want that: https://github.com/kripken/emscripten/wiki/WebAssembly#binaryen-methods

• Running wasm file on Node doesn't work because latest Node doesn't expose Memory property of WebAssembly object
• Looking into wast file I don't see "Hello, Emscripten!" string anywhere
• Then I've created index.html with such content:

<script type='text/javascript'>
 fetch('hello.wasm')
  .then(response => response.arrayBuffer())
  .then(buffer => WebAssembly.compile(buffer))
  .then(console.log)
</script>

and ran Python's SimpleHTTPServer. Running this in latest Chrome Canary with WebAssembly enabled doesn't do anything while not throwing errors and showing an empty wasm module object.

So I'm not sure if Rust code cross-compiles to correct wasm module at all for now (using rustc 1.14.0 and emcc 1.37.0 / 1.37.1). Am I doing something wrong?

[kripken]

In general you can open an issue on the binaryen tracker (often it's the right place, if not that's fine too). Regarding your issues:

• Yes, node.js doesn't have wasm support yet. You can use SpiderMonkey or V8 shells (very latest versions), or nightly/canary versions of the browsers. Or, enable the interpreter, which will work everywhere.
• You should see the string in the wast, that is surprising. It should be in the memory segments, near the top of the file. Are there no memory segments? Earlier binaryen versions didn't use them, but we should now.
• You can't load the wasm by yourself - it isn't standalone (there is work towards standalone wasm, which perhaps rust can use eventually). The JS code knows how to load it and run it. Try to just create an html page with <script src="the.js"></script> in it.

[chicoxyzzy]

Just created WebAssembly/binaryen#867

[chicoxyzzy]

Update: cross-compiling to wasm works as expected

[CryZe]

There's one massive problem when compiling Rust to WASM which made it fail in about half the Advent of Code puzzles I tried (and a lot of other projects including my current large project). It's the fact that Rust generates auto vectorized code which then fails in the asm2wasm conversion step as it doesn't support vectorization yet. It is impossible to turn off auto vectorization at the moment as it seems. Neither the emscripten flags nor the Rust flags seem to affect it. So this definitely needs to be fixed.

Here's the binaryen issue for it: WebAssembly/binaryen#855

Update: Playing around more with this, this seems to just randomly work now if you pass the no vectorization flags?!

Update 2: Nvm, it still doesn't work in my larger project. This is just super odd. I guess a lot of commonly used crates do manual vectorization which you can't just turn off with flags?!

Update 3: Turns out it's not any crates that cause this. This is definitely either rustc or emcc generating vector instructions that you can't turn off.

[aidanhs]

@kripken

as currently it is not yet stable and produces larger/slower code

I assume the llvm parts are mostly similar, so you're referring to the emscripten asm.js optimizer passes? Which is a partially solved problem if documentation includes the three commands necessary to use ayzim (though this would require the converter to not 'finalize' the asm.js).

[kripken]

@aidanhs Yes, LLVM IR optimizations are identical - the difference is only in "backend" opts. But the asm.js optimizer isn't necessarily used in this case. On the LLVM backend side, LLVM does the backend opts; on the asm2wasm side, backend opts may be done by either the asm.js optimizer (from emscripten, or in principle could be ayzim) or wasm optimizer (from binaryen), or just with the wasm optimizer by itself. The last option is currently enabled by default, as it has some benefits, although the asm.js optimizer is automatically run if the code is also meant to be runnable as asm.js as an option. (I have a blogpost in the works about this stuff.)

[aidanhs]

Right, I see the issue more clearly now - binaryen is good, but the llvm wasm backend doesn't contain all of binaryen.

If a wasm file generated by the llvm backend were run through the binaryen optimizer, would it 'converge' to be similar to a file generated via the asm2wasm route? If so, it might be reasonable to offer the binaryen optimizer as a separate tool (or ayzim, when I port it). Depending on what the rust team prefer this optimizer (C++ or rust) could be inside rustc as a strange post-compilation pass, with the promise that the code will disappear as the llvm wasm backend gets more mature.

Regarding stability, I'd be in favour of the wasm llvm backend regardless of optimizer thoughts - more exposure will make the backend better, and the rust users giving it a go as a first class target will help make sure that it's robust come wasm 'release'.