Midarr Labs
Replies: 40 comments
-
|
Thanks for opening this discussion. I think this is definitely a topic worth talking and exploring more about, and as you’ve mentioned Midarr now has a half baked solution for this. You’ve brought up a few interesting points to discuss:
|
Beta Was this translation helpful? Give feedback.
-
ffmpeg jellyfinWhat is the benefit of this over say just ffmpeg itself? A quick look through the repo it seems jellyfin has correlated only the tools they need for their transcoding pipeline. Unless I’m missing something? |
Beta Was this translation helpful? Give feedback.
-
|
I’m all for not reinventing the wheel and have looked into other external / decoupled solutions like Go Transcode before doing a basic Midarr implementation. Go Transcode not only provides the ffmpeg / transcoder implementation, but also the “glue” or interface Midarr can consume for transcoding. |
Beta Was this translation helpful? Give feedback.
-
|
If Go Transcode is a viable option to get us up and running quickly, but also achieve load balancing across Midarr instances. All that we’d need really is the Midarr implementation for it. |
Beta Was this translation helpful? Give feedback.
-
|
I just picked jellyfins ffmpeg because I figured it would be a fork of ffmpeg geared towards transcoding various different types of media so many nothing too special there. go-transcode seems interesting as well! |
Beta Was this translation helpful? Give feedback.
-
|
Would you be willing to test out Go Transcode and see if it’s a good viable solution? It’s come a long way since I last looked at it. It didn’t have VOD support back then, which I see it does now. |
Beta Was this translation helpful? Give feedback.
-
|
What exactly is Midarr lacking in its albeit basic transcoder at the moment? How does it play your media that previously you said didn’t have any audio? If you had a wishlist for Midarr transcoding, what would that be? |
Beta Was this translation helpful? Give feedback.
-
That's a pretty hard question to answer because I am very much in love with Kubernetes for managing containers. It would be awesome if Midarr could have a "transcoding" service which would be a dedicated number of "warm" containers just for transcoding. To get deeper into the weeds it would amazing if Midarr could speak to the kubernetes API to create/destroy pods on demand for transcoding while also having "warm" transcoding pods already available. As far as I know in standard docker-compose land, or hell even using a thing like Portainer this would not be possible. Kubernetes is amazing at automation like which is why I am so in love with managing containers with it. A Midarr Kubernetes operator would be amazing if you wanted to learn more about Kubernetes 😄 but I understand this is a huge change and would require major refactoring. You did ask for a wishlist though 🤣 |
Beta Was this translation helpful? Give feedback.
-
|
Yeah Kubernetes is amazing I agree! That’s quite a Wishlist 🤣 a Midarr K8s operator sounds interesting, but is out of scope for the Midarr app itself. I created exstream as the upstream transcoding service Midarr currently relies on. Packaging this service up in its own Docker container would be a start - and we’d be able to scale this Plus by all means - if you’re keen to create a Midarr K8s operator, you have my full support! |
Beta Was this translation helpful? Give feedback.
-
|
Maybe an operator could be done in the long term. Breaking out the transcoding from the main app would be very neat in any way it could be done. That would really set Midarr apart from the other players in the game like Jellyfin and Plex which are behemoth monoliths. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah wouldn’t take much to package up exstream, I’ll just need to figure out the public API for it 😁 I’ll look into it after v3 release. Let’s keep this conversation going though - a lot of great ideas already! |
Beta Was this translation helpful? Give feedback.
-
|
I just had this idea and came here to find the discussion about transcoding already going, but I figured I'll share it anyways: Midarr's approach of leveraging existing services that people are already running is very nice. If that could somehow be applied to transcoding too, for example by hooking into something like Tdarr, that would be pretty cool. I don't know whether Tdarr (or other software like it) would support something like that at all though. If that's not possible, I agree with onedr0p that breaking out the transcoding to exstream or another app running externally would be very nice to have. Over at Immich we don't need to do any live media handling, but we still have all the processing done in a separate container which keeps the main UI/API nice and responsive and makes scaling easier to reason about too :D |
Beta Was this translation helpful? Give feedback.
-
|
Tdarr not being open source is a full stop for me. |
Beta Was this translation helpful? Give feedback.
-
|
I wasn't aware of that, but I agree! I just used Tdarr to illustrate my idea, I believe there are other similar applications out there that are open source. |
Beta Was this translation helpful? Give feedback.
-
services:
midarr:
container_name: midarr
image: ghcr.io/midarrlabs/midarr-server:latest
environment:
- EXSTREAM_URL=http://exstream
exstream:
container_name: exstream
image: exstream:latest
volumes:
- /path/to/media:/mediaJust noting this down for more discussion. If we packaged up exstream into its own image - and have Midarr reference it for all streams, this would be a breaking change. I'm not too keen on having Midarr default to itself, then reference exstream optionally (opt in to exstream). What do you think? |
Beta Was this translation helpful? Give feedback.
-
|
This is maybe more for implementation details of Exstream, but https://membrane.stream/ could be worth taking a solid look at. |
Beta Was this translation helpful? Give feedback.
-
|
Awesome thanks for explaining that. I was initially thinking of the proxy option you mentioned. I would prefer Midarr be as loosely coupled as possible to Exstream. I’m leaning more towards Exstream just being a standalone HTTP service that anyone could use. Midarr would just implement the API. |
Beta Was this translation helpful? Give feedback.
-
For this part - does Midarr really need to know those specific details? I’m just asking because I’m unsure. If you scaled out Exstream to say 10 instances / containers and had Traefik load balance across them - could Midarr not just reference that single Traefik URL? Midarr wouldn’t need to know there are potentially more than 1 instance. Am I on the right path or way off? 😂 |
Beta Was this translation helpful? Give feedback.
-
It could but what happens when there is 10 transcoding requests and they happen to land on the same exostream container? This is why I'm thinking midarr has to be aware of what work is happening on which exostream container and coordinate accordingly. |
Beta Was this translation helpful? Give feedback.
-
|
Ah true!! If that could potentially happen, then yes Midarr would need to be aware. Might be worth me doing some discovery work around that too maybe. Just to validate our assumptions 😁 |
Beta Was this translation helpful? Give feedback.
-
https://membrane.stream/guide/v0.9/packages.html They don't seem to support h265 (yet?) 😢 |
Beta Was this translation helpful? Give feedback.
-
Damn, I missed that :/ After looking at Exstream a bit more, I think Membrane might be overkill here anyways. I laid out the architecture q to a few friends and together we came up with these options:
With approach 2, Midarr needs to regularly re-fetch the DNS records to make sure it's aware of instances being created/removed/etc. The Exstream API should probably also have some endpoints for its status (eg system load, health, whether it's shutting down) that Midarr keeps track of. |
Beta Was this translation helpful? Give feedback.
-
|
This is awesome thanks for this! |
Beta Was this translation helpful? Give feedback.
-
For this option does this register a single instance? |
Beta Was this translation helpful? Give feedback.
-
This we could definitely work towards. A few moving parts to break down into smaller deliverables, but I think the payoff would be huge! Being able to scale Exstream out with Midarr aware! |
Beta Was this translation helpful? Give feedback.
-
Something like a Prometheus exporter / stats endpoint? 🔥 |
Beta Was this translation helpful? Give feedback.
-
|
I’ve started working on a MVP release for Exstream already. Just for the MVP I’ll get a minimal functional version working - then we can build these features on top and over the next iterations. |
Beta Was this translation helpful? Give feedback.
-
|
I’ll add a roadmap for Exstream too so we can prioritise these features for releases. |
Beta Was this translation helpful? Give feedback.
-
This would be one instance of Exstream with a sidecar script that registers that instance, but that approach can be repeated to register multiple instances (each with their own sidecar).
That'd certainly be good to have, and if it has the right metrics then Midarr can use it to monitor the instance too. The goal there is that Midarr knows how the Exstream instances are doing load-wise etc, so it can make smarter decisions about where to assign new streams. |
Beta Was this translation helpful? Give feedback.
-
|
I'm going to move this into a discussion, as it's quite a broad topic. We can create issues tied to the discussion as we progress. |
Beta Was this translation helpful? Give feedback.
-
This is mainly also on the topic of transcoding, I know you have some opinions on Midarr and transcoding but this is to bring up a discussion transcoding and how it could be implemented if you were to ever think about adding such abilities.
The way Midarr works currently is that transcoding is not really supported, anything outside x264 most likely will not play. I wonder if it is worth the effort to use jellyfin's ffmpeg as a library to integrate transcoding in general.
On the topic of distributed transcoding
Given you agree that transcoding is on the slate for possibly being implemented...
I really love that I can scale midarr to
nnumber of pods in kubernetes, however there is no way to "load balance" transcoding requests so that an overworked container will not pick up new requests for transcoding.These are just questions to chew over on future ideas of Midarr.
Beta Was this translation helpful? Give feedback.
All reactions