IMSC/TTML support

[ArthurKnoep]

Description

Hi guys,

At the moment only WebVTT subtitles are supported for both DASH and HLS media source. However WebVTT is a very limited file format in terms of functionality, that is why a new standard for the web tends to emerge: IMSC (Internet Media Subtitles and Captions). To explain very briefly the concept, IMSC is pretty much HTML but with time tags so that only some XML elements appear based on time. This allows for custom text styling, custom text placement and pretty much anything you can do with basic HTML/CSS. (There is a demo on the MDN for those interested: https://developer.mozilla.org/en-US/docs/Related/IMSC)

IMSC can be presented under two distinct format:

• it can either be a text file under the following content-type: application/ttml+xml
• or it can be segmented as an fMP4 file where the xml content is stored under an ISOBMFF encapsulation using the following content-type: application/mp4

DASH has already official support for it, per the section 6.4.4 of the specification: https://dashif.org/docs/DASH-IF-IOP-v4.3.pdf.
For HLS, Apple has announced support for it in 2017 for their hardware, however the updated RFC is still in draft: https://datatracker.ietf.org/doc/draft-pantos-hls-rfc8216bis/

For a personal project of mine I need to playback video files with IMSC subtitles, my videos are available both with HLS and DASH (for DRM purpose, Safari only supports HLS for DRMs). My initial plan was to create a custom plugin that would somehow extend VHS so that it can handle IMSC in addition to WebVTT. However it appeared to me that subtitles handling was too much intricate within VHS code base to be expanded by a VideoJS plugin.
At the moment, the only solution I have is to remove subtitles from my HLS and DASH manifest, and add them as external text track for VideoJS. Then, I use a plugin I wrote that provides a custom source handler so that it can play them. This is not an optimal behavior as it requires to add another endpoint to my API to list the available subtitles and adds additional latency and frictions for initializing a playback.

My question is the following, would there be any motivation for having support for IMSC subtitles within VHS ?

Note that I could be up for the development but as I'm not familiar with VHS this will probably take me a lot of time without guidance and also I don't want to work on something that will not be accepted.

Sources

I can't share a source publicly available as I'm only conducting tests on my local laptop at the moment, however I can share a DASH AdaptationSet that loads an IMSC subtitle (the IMSC subtitle in question can be found here: basic-expanded.ttml)

...
<AdaptationSet id="2" contentType="text" mimeType="application/ttml+xml" lang="eng">
	<Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
	<Representation id="2" bandwidth="1000">
		<BaseURL>basic-expanded.ttml</BaseURL>
	</Representation>
</AdaptationSet>
...

Results

Expected

When playing a DASH or HLS playlist with IMSC subtitles, I expect to see my subtitles displayed on the player.

Error output

Subtitles are handled by vttjs, so it does nothing with this warning message:
video.js:210 VIDEOJS: WARN: Error encountered when parsing cues: Malformed WebVTT signature.

videojs-http-streaming version

videojs-http-streaming 3.6.0

videojs version

video.js 8.6.0

Browsers

• all

Platforms

• all

Other Plugins

none

Other JavaScript

none

[welcome]

👋 Thanks for opening your first issue here! 👋

If you're reporting a 🐞 bug, please make sure you include steps to reproduce it. We get a lot of issues on this repo, so please be patient and we will get back to you as soon as we can.
To help make it easier for us to investigate your issue, please follow the contributing guidelines.

[adrums86]

I believe the motivation to further expand the text support within video.js/http-streaming is becoming greater as the community continues to grow. Generally any improvements or support for functionality such as this will be accepted as long as it meets the basic guidelines. Although the core team may not be currently prioritizing IMSC/TTML support, your contributions would be very much welcomed! We are always happy to help answer questions and offer guidance when we can. For this specific feature, work in the (DASH and HLS) parsers, mux.js, and finally here in VHS would likely be required for FULL support. That said, we would be excited about any meaningful contribution to any of these projects and if the motivation for other members exists as I suspect it does, you would likely get some other contributors to your work with you as well.

[gkatsev]

IMSC support has just not bubbled up to the top because the vast majority of the streams we had dealt with were WebVTT.
Though, rather than parsing IMSC ourselves, I'd recommend we integrate with imscJS instead. However, given it's file size (with a full SAX parser for the XML), I'd suggest that we use it if it's available on the page (or provided via options, i.e., DI). Though, even if imscjs updates to use DOMParser, it may still be better to keep it around via DI only, since it'll still be around 50KB.