[Proof of Concept] Video Export using Web Codec API #178
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR explores a way to use the Web Codec API to export video. This is not only faster than using the JS implementation we’re currently using, but also opens up more export formats (including H.264 codec in MP4 container).
This is in the early stages of exploring. Video formats, codecs and container are a fairly complex topic and I feel like I'm just starting to understand.
What this does
VideoWriter
class to mechanic core, replacing the previous WebmWriteranimationFormat
to set the desired exportwebm
andmp4
(defaults to webm, as we have a fallback for non-supported browsers)animationBitrate
for exportWhat could be done
mp4
andwebm
Test Driving
fillAnimatedCanvas
is set up to export MP4 videoThere’s probably a million more things to discuss that I can't even begin to think of, as video-formats, containers, and codecs are quite the rabbit hole. I tried to leave extensive code comments to document what is going on and why certain decisions were taken. But please challenge everything I've done 😊
Caveats
Not all browsers come with full support for the web codec API, Firefox for example has no support for the Web Codec API currently at all. But there's also differences in Chromium based browsers: Chromium (as it's used on Linux or in headless scenarios) only comes with built-in support for open codecs and lacks proprietary codecs. This means no h.264 in Chromium. Google Chrome however has full support including proprietary codecs.
However, when using Chrome, the licensing of h.264 is covered by Chrome's license and we’re good to use it.
Alternatives
I did a test drive using
ffmpeg-wasm
locally and it works across browsers (including headless Chromium), so this could be a feasible alternative to relying on the Web Codec API. Howeverffmpeg-wasm
comes at the cost of downloading ~25MB of WebAssembly code to the user's browser first.As h.264 is a proprietary codec there is some confusion around whether we'd be allowed to embed it (in the form of ffmpeg-wasm). There is some indication that we might don’t want to do this (or that we could do it, but need to inform our users about this, so they don’t get into trouble).
For references see: