Add H.264 decoding by utilizing OpenH264 from Cisco

[torokati44]

This is very much a WIP. Progresses #723. Basically pushing only for visibility. Will partially replace #12539 (first only on desktop, in the future, using WebCodecs, entirely).

AFAIK this is the only legal way (without us or our users having to pay royalties to MPEG LA VIA LA) to do this with reasonable portability on desktop, but IANAL. See this little explainer video: https://vimeo.com/79578794

The code is somewhat based on the openh264-rs crate, see ralfbiedert/openh264-rs#43.

YUV->RGB colorspace conversion is done by our trusty handcrafted code, like for H.263 and VP6[A]. This means that it will always assume BT.601 coefficients, even though it could theoretically, possibly be BT.709 or something like that. But the difference is probably subtle enough anyway.

TODOs, in no particular order:

• Add a configure_decoder (or similar) method to the video backend/decoder traits. Currently the preload method is misused to pass the "extradata" to the decoder.
  • Done.
• Figure out how to not create colliding Indexes for proxied and handled videos in the ExternalBackend.
  • Added an enum to store in the new backend's arena. TODO: Add a warning comment.
• Optimize copying YUV data out, and possibly AVCC->AnnexB format conversion
• Handle downloading and hash-checking of the decoder shared libraries for the current platform/arch.
  • Done. TODO: Needs to do error handling better for proper fallback and hints for the user.
• Add the Cisco license text, display it somewhere in the player.
  • Done. The new backend will write it into the current working directory as OpenH264-license.txt upon creation.
• Add a command line option to disable the decoder (as per Cisco's license terms, I hope this will be enough)
  • Done: --enable-openh264 [true|false]
• Switch to preferences for the enablement toggle
• Figure out how, or if, to test this (command line flag for implicit agreement to fetch the decoder library on every run?)
  • Download on first run without confirmation, even in CI.
  • For tests, this should be opt-in, just like "video" and "renderer" in general.
  • Command line flag will be only for opting out.
• Write up a method for updating OpenH264 versions in the future.
• Check if undoing the "start code emulation prevention" for the decoder config (extradata) is needed, and undo it if so.
  • Left this as a TODO comment for now...
• Look at the presentation time offsets as provided by the FLV demuxer (later the MP4 container), and the fields in the SBufferInfo struct from the decoder.
• General code cleanup, including properly working features to disable this, comments, and error handling.
  • Almost done!
• Testing, testing, testing...
  • There is a visual test added, which is something...

TODOs for later:

• WebCodecs implementation for web. Consider whether MSE plus a hidden <video> element would be a feasible fallback - probably not, but who knows.
  I plan to put this into the same ExternalVideoBackend added here. Also with the "other ideas" below. Spares us from duplicating the proxying logic. The Android backend will have to be different though, as it's somewhere else.

Somewhat less closely related TODOs for later:

• MP4 (F4V) support: video: Add support for playing F4V (MP4) files #14655
• AAC-LC decoding support (already available in Symphonia).

Other ideas:

• Different, specific implementations for different platforms: Video Toolbox on macOS, Microsoft Media Foundation or DXVA[2] on Windows, VA-API (or VDPAU...) on Linux, and MediaCodec on Android. Maybe Vulkan Video anywhere applicable. (Loading FFmpeg or GStreamer on Linux would be iffy due to uncertain licensing of the used decoder.)

[torokati44]

This is what I use to test this at the moment:
f4vplayer_h264_flv.zip

I hope it's okay to upload here. Despite the name, it now loads an FLV file. It's originally from here: https://permadi.com/2010/02/flash-playing-flvf4v-videos-with-script/

Of course, the decoder library still has to be obtained manually at the moment.

[danielhjacobs]

Will partially replace #12539 (first only on desktop, in the future, using WebCodecs, entirely).

The VideoDecoder part of WebCodecs (and many other APIs) are only available in secure contexts, right (they're also not yet available in Firefox but I'm sure that will change)? Will this be the case in insecure contexts too? I suppose browsers are moving away from allowing insecure sites at all.

[torokati44]

not yet available in Firefox but I'm sure that will change)?

Certainly, since it already works if enabled in about:config, it's just not yet stable/complete.

The VideoDecoder part of WebCodecs (and many other APIs) are only available in secure contexts, right

I have no idea. Anyway, we can keep the current solution if VideoDecoder is not available for whatever reason.

[torokati44]

I'm not entirely sure how enablement should be handled w.r.t. #16310... 🤔 ...as this is entirely desktop-specific.

[sleepycatcoding]

I'm not entirely sure how enablement should be handled w.r.t. #16310... 🤔 ...as this is entirely desktop-specific.

Maybe this should be in preferences instead and have an option to also toggle it in the UI?

[torokati44]

Maybe this should be in preferences instead and have a option to also toggle it in the UI?

Hahah, yeah, you're right, there just weren't preferences yet when I started working on it, and they didn't come to my mind just now... 😅

[torokati44]

Made the toggle a preference now, instead of a command line option.

[torokati44]

There's now a bit of a back-and-forth between video/external: Add OpenH264 decoder and desktop: Add a preference to enable the OpenH264 decoder regarding license handling, but it doesn't bother even me that much to go through and rebase it out...