feat: initial closed captions support

[arthurdenner]

Context

I was recently playing around with FFmpeg to embed SRT files as closed captions, merging them into the final video as subtitle tracks.
Regarding FFmpeg, it's somewhat easy to do but I've found a couple of issues.
I'm opening this draft PR so that others can have a look and we can start a discussion about an API for this if we see fit for the project.

Changes

• Reduced the number of frames for the remote-video demo for faster feedback loop;
• Added an SRT file with 3 subtitles;
• Resolved the path for the subtitle and added it to the list of FFmpeg inputs;
• Added the FFmpeg flags to embed the subtitle into the final video;

Notes

• The final video has the duration of the subtitle, which is wrong and longer than the duration of frames;
• 2:s is used before the subtitle is the third input. If there were more, it would change accordingly;
• mov_text is needed to encode the SRT file and it's the only subtitle encoder supported in an MP4 container
  • Wiki reference
  • Stack Overflow reference

Demo

Here is a video with the output of this branch/PR.

PS: The video is cut so it doesn't show any info from my VLC but I hope it gives you a good picture of the result, including the duration bug.

remotion-cc-demo.mp4

[JonnyBurger]

This is pretty cool!

Few thoughts:

It would be cool if we can declare the captions using React markup, using e.g. <Caption /> components.

We can also write a function that loads .srt files and converts it to <Caption /> elements, also allowing transformations inbetween.

There are two ways people use captions: Either embedded in the video file, and you can turn them on / off or switch languages, or burnt into the video. While the second type can already be done using Remotion, maybe we can provide some helpers too.

Twitter wants publishers to upload a separate .srt files, so I feel like we also need to provide an option to output a .srt file as a secondary output file alongside the actual video.

This is a great proof of concept showing that all of this is possible, so thanks a lot for making it!
Also linking this to #16 where a lot of discussion already started. I will from time to time contribute to this PR and hopefully we can turn it into a real feature over time!

[JonnyBurger]

Bug possibly in macOS: I cannot select subtitles when opening the video in QuickTime, however they do show up in Preview (when I press space bar on a video)

[arthurdenner]

Thank you for the feedback.

I was able to select the subtitle in QuickTime, I'm not sure what happened on your side.

I think that a <Caption /> component is a good start.
I haven't explored it yet but I imagine it would:

• Refer to local or remote files;
• Allow us to pass metadata such as language to the caption (and fix the "Unknown language" in the screenshot above - I tested locally)
• Dynamically compute the FFmpeg filters

I'll start to look into it and update this PR.
Feel free to contribute to it with ideas and code as well.

[JonnyBurger]

I think QuickTime is sometimes a bit buggy, but probably nothing we can do about it. Like the idea of being able to load subtitles remotely! And agree with multilingual support and automatic transformation into FFMPEG syntax.

Myself I am probably going to be busy with Lambda for the next month at least, and would want to pick up other big projects after that.

[arthurdenner]

@JonnyBurger, based on how assets (audio and video) are implemented, I implemented a draft of a <Caption /> component.

It only supports remote captions and SRT files for now and it downloads them during render - just like remote assets.
The FFmpeg filters are computed based on the number of captions + the assets count - to calculate the offset of inputs.

It supports some metadata properties (language and title) to improve the output video.
Here are some screenshots of the demo in this branch - check the instructions on packages/example/src/RemoteVideo/README.md.

<Caption
  language="eng"
  src="http://127.0.0.1:8080/subs.srt"
  title="Some title"
/>
<Caption src="http://127.0.0.1:8080/subs_alt.srt" />

Screenshots

Please provide feedback on my decisions and work so far when you have some time.
I know Lambda is the focus right now.

Best regards! 😃

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
remotion	✅ Ready (Inspect)	Visit Preview	Aug 29, 2022 at 5:53PM (UTC)

[JonnyBurger]

Sorry for taking a year to reply! 😅
I've merged over 4500 commits from main and got the proof of concept working again.

Some thoughts on how to proceed:

• <Caption> could either act as "baked in" subtitles where some text is rendered or as a track that is embedded but can be toggled by the video player. Let's make a proper distinction between it, maybe by splitting it up into two components?
• For baked in subtitles, it would be nice if you could style them using CSS.
• For non-baked subtitles, it would be nice if you could see them in the preview.
• It would be nice to create your Subtitle track in React, something like:

<>
	  <Sequence from={0} durationInFrames={20}>
		  <Subtitle>Hello World</Subtitle>
	  </Sequence>
	  <Sequence from={20} durationInFrames={40}>
		  <Subtitle>Second subtitle</Subtitle>
	  </Sequence>
</>

Being able to take advantage of existing primitives like Sequence and Series would be nice.

• I refactored the subtitle file being pulled in so no extra server is needed. Right now to FFMPEG it is passed -i http://localhost:3001/public/subs.srt which unintentionally works, it would be better to download it like the audio files.

[arthurdenner]

The filters for rendering subtitles were broken, I fixed them in my commits.
I also merged main again and restored some comments because problems from the first version are still present:

• The final video has the duration of the subtitle, which is wrong and longer than the duration of frames;
• mov_text is needed to encode the SRT file and it's the only subtitle encoder supported in an MP4 container

I haven't addressed any of your comments yet but here are a few notes/questions:

1. With "baked in" subtitles, did you mean hard/burned subtitles?
2. I'm not sure how much you can style a subtitle, I think it's limited and doesn't relate to CSS
3. Regarding the preview, the POC handles SRT files, the <track /> element expects VTT files
4. Regarding creating subtitles, I think it's a good idea but maybe in a future PR.
   We should set some scopes and milestones to progress step by step IMO.

[JonnyBurger]

Thanks for fixing!

• Is it possible to trim the subtitles e.g. using https://superuser.com/questions/1651181/ffmpeg-hard-subs-and-trim?
• What is the disadvantage if we have to use mov_text?

By baked in subtitles I mean burned subtitles yeah. The styling would only work for burned in subtitles of course.
I agree though that we can start by shipping something small - let's say only VTT subtitles as long as we keep the API flexible that we can improve on it in the future.

[arthurdenner]

• I don't know about trimming, haven't explored it but it seems complex?
• mov_text is rather a limitation, it seems to be the only way to have SRTs in MP4s.

[Just-Moh-it]

+1 on adding this component. Need the 'baked-in' subtitles just as much as the video component for a project we've been working on