[Feature Request] WebVTT subtitles support

[jamsch]

What is the need and use case of this feature?

This is a wild shot, but would it be feasible adding WebVTT support in RNTP? Nothing in the RN space seems to support the following features:

• Podcast-style audio, where the conversation transcript can be shown
• Song lyrics, where the lines of the song can be highlighted during playback for users to sing-along.
• Audio-based lessons (e.g. languages), where combining listening & reading would be more informative.
• Accessibility, where some users may be hard of hearing and may need to read instead.

Describe the ideal solution

• Native integration in Kotlin/Swift which attaches a subtitle source to the player from a WebVTT file path or blob.
• A native listener on the player for cue changes, e.g. for Android this would be Player.Listener@onCues
• A module export for registering a listener for cue changes, returning something similar to the cuechange event so that the user can access the current track cue.
• A React hook that builds on top of the event listener API for basic use cases inside React components.
• Alternatively, some sort of SubtitleView component export from the library, however I don't know how this would play well with iOS.

Describe alternatives you've considered

1. Just rendering everything inside a web view. This is essentially what we're doing on our web app:

<audio src="https://website.com/path/to/audio.ogg">
    <track
      src="https://website.com/path/to/subtitles.webvtt"
      kind="subtitles"
      src-lang="en"
      label="English"
      default
    />
</audio>

<script>
  document.querySelector("audio > track").addEventListener("cuechange", (event) => {
    console.log("active subtitle: ", event.target?.track.activeCues[0]?.text);
  });
</script>

2. On the mobile app, I'm instead parsing out the WebVTT file contents in JS, such as the following:

export type WebVTTCue = {
  startTime: number;
  endTime: number;
  text: string;
};

export function parseWebVTT(input: string): { cues: WebVTTCue[] }

// ...
const { cues } = parseWebVTT(`WEBVTT\n\n1\n00:00:00.960 --> 00:00:03.000\n....`);

And using a simple binary search to locate the current cue:

function getCurrentCue(cues: WebVTTCue[], timestamp: number): WebVTTCue | false

However the issue with this current approach is that I'm finding that the timestamps (at least with the expo-av lib at the moment) are extremely unreliable and can be seconds out of sync with the actual underlying audio progress (even with no JS frame loss + 250ms reporting intervals).

Additional context

After experimenting with integrating WebVTT in Android Media3 ExoPlayer (using Expo Native Modules) about a year ago on this issue, I managed to get it working pretty good, however the solution was extremely hacky (due to my limited knowledge of the Android ecosystem) and I haven't yet tried integrating it on a mature AV project such as this one, or expo-av.

How I can Help

I'll likely have some free time soon to work on this again for Android and iOS, however my efforts will be in integrating this feature in a simple standalone player in a separate repo, just so that I can get a proof-of-concept out. I have no expertise in Kotlin/Swift so there'll need to be an experienced native developer to be able to integrate it in to this library.

[lovegaoshi]

this is a music library though, this is more relevant for the RN-video community (and is being worked on):
TheWidlarzGroup/react-native-video#3541
isnt .lrc more relevant for music? and there are solutions for this
RNTP really just plays music in the background, wont and shouldnt give a webview

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.