The goal of Scythe was to be able to capture the last N seconds of a screen/app and add audio commentary over the top. The source idea was to improve my Overwatch play by capturing my deaths and laying on top my thoughts for a more immediate game-play review.
It's written using Electron, TypeScript and FFMPEG, designed to be an overlay widget, similar to Discord's overlay
This is not a finished project and at present I will not be returning to it. As you'll find below I simply could not get the screen recording to work properly within the technical bounds I set myself (staying within electron) and wish to spend my time exploring other ideas and topics. I've opened sourced this project to document my attempts and as a reference for the vairous techniques used throughout.
I've not packaged this up but you can run it locally.
yarn installyarn run start- Select the screen and mic to use
- Hit
f8to commit the last 10 seconds and start recording audio - Hit
f9to close the audio recording and finish up - You'll find the MP4 in
/output
The screen recorder needs to be "always on", keeping the last N seconds in memory such that those seconds are readily available on command, like the PlayStation share clip. All my attempts revolve around the use of the MediaRecorder API. An instance of MediaRecorder takes a MediaStream and records it into webM file in the form of a Blob (or several as we'll see).
Start up a MediaRecorder, setting it to output a Blob of recording once a second. This Blob is pushed onto an LRU queue with a fixed size of N (N * 1s = Ns of recording). When requested, the contents of the queue was stitched together by Blob concatenation then saved to disk.
Result; Corrupted webM files mostly. The very first Blob output of a MediaRecorder contains metadata required to form a valid webM file. So once the app had ran for N + 1s (so the first Blob had been evicted from the cache) the output was a webM file with no metadata. VLC could kinda of play it but with very weird behaviour.
Improvements; If I could figure out how to manually create the metadata for a webM and stitch it in front of the Blobs in memory, this might of worked.
Another single instance of MediaRecorder but this time saving each Blob as an individual webM file and using FFMPEG to stitch them together.
Result; Due to the missing metadata in each Blob, FFMPEG would fail with an "invalid data" error.
Improvements; Again the metadata, though if that was solved, attempt 1 would be a better solution still.
Multiple instances of MediaRecorder. Start a new recorder every second and stop it after N seconds. Maintain a latestBlob variable which each recorder overrides. This attempt was to make sure I had the metadata in each Blob but it was to be much more expensive (N^2 frames in memory).
Result; This worked... when recording basic apps like VS Code or Chrome. I booted up Overwatch, started recording and oh my. The process went up to 88% CPU usage and my laptop went into meltdown. Everything would come to a halt before more than a second of recording could take place so there was no output. This was with a very low N of 5-15. I also played with the recording dimensions and frame rate.
Improvements; This was just a very inefficient way of doing it but it's also the closest I got. I'm not sure what else I can try here.
Take 30 screenshots every second, pass them to FFMPEG to stitch together as a video.
Result; Didn't even get to the FFMPEG part. Within 5 seconds I'd created 142mb of images. That number went down with tweaks but wasn't sustainable for say, an N of 30.
Improvments; This was a pretty lame attempt I'll admit. I could probably keep a lot of images in memory and batch save to disk (I'm using FFMPEG via the command line).
Back to a single instance of MediaRecorder but this time try to create the metadata. I created a "head Blob" but starting and quickly stopping another MediaRecorder before starting the main one. Then on stitching (like attempt 1) I'd pop this head Blob in front. I also tried creating the head Blob at the point of capture too.
Result; This created a valid webM but the metadata was seriously messed up. A 5s clip would be actually be 17s long with the first 12s being (I assume) the frame in the head Blob.
Improvements; Being able to create valid metadata is a recurring theme. Again if this was solved, attempt 1 would be better.
Screen recording is a solved problem in other platforms. Windows provides an API with per-frame callback and NVIDIA also have an SDK for some of their GPUs. If I'm to come back to this I'd try the Windows API next using cross process comms to talk to the electron app (which would just become a UI tbh).
Or maybe I've missed something in my attempts. Maybe there is a different set of config for the MediaRecorder or a different recording API entirely.
Despite the recording failing there are other little bits in this project I'm quite pleased with.
fp-ts; This is my first usage of fp-ts and I very much enjoyed it, having HKTs in TypeScript is a joy. A number of the platfrom methods around video/audio are asynchronous and can fail. fp-ts/TaskEither was a wonderful abstraction over these concepts, especially once I discovered the tryCatch function.
Validation; I'm also fond of the form validation in config.ts (it could do with a large refactor) which combines applicative validation with TypeScript's annoymous type equality to produce nice compile time guarantees. For example, in tryBuildConfig, if I forget to add a validator for a form property then the anonymous type won't be equal to the resultant type (by missing the property) and will not compile.
most; The application is stateful with a fair amount of async and events flying around. I used a flux-like architecture via most to orchestrate this. Inspired by redux-observable, I used Streams to chain the async handling.