[ENHANCEMENT] Better/Smoother playback from rtc stream

[stevelord]

Describe the bug
While playing videos, there's a general jerkiness every few seconds.

To Reproduce
Steps to reproduce the behavior:

1. Configure a CPU-optimized Digital Ocean droplet with 8Gb of RAM in your closests datacentre
2. Install Neko as per instructions
3. Log in, go to this trailer, play it, full screen it, full screen your browser.
4. Note the jerking every few seconds, with jerking and tearing particularly noticeable in camera pans

Expected behavior
Video playback as smooth as a baby's bum

Desktop (please complete the following information):

• OS: MacOS
• Browser: Chrome, Safari
• Version FF 74.0, Chrome 80.0.3987.149

Additional context
I'm not sure if this is a problem with X11, or X11VNC but I've tried on different systems with different browser combinations and it's consistent. Happy to look into it if someone can give me some pointers.

CPU load is fine, RAM is fine. There could be a sync and framerate issue or something can't keep up.

[nurdism]

I've noticed some stuttering myself occasionally. I believe its the encoding/gstreamer that is lagging behind when attempting to get low latency rendering.

You can see here where the settings are passed to gst https://github.com/nurdism/neko/blob/master/server/internal/gst/gst.go#L87 I've been able to get good smooth video but at the cost of the latency being way higher. You more than welcome to mess with it and see if you can get something better going.

[stevelord]

Do you know what settings you used for smooth video? I'm trying to build at the moment but I'm finding the whole VS/docker setup a bit difficult as I'm on a Mac. Working around it, but it's slow.

[GigaFyde]

Cannot exactly reproduce on my dedi.

What resolution settings in neko are you using?

[GigaFyde]

Also, neko is incredibly reliant on the udp stream that carries the video data to your client, if that connection experiences hiccups the stream will stutter.

[stevelord]

@GigaFyde it's all the default settings from the build.

I did find something interesting last night. I built a derived docker container launching VLC instead of Firefox or Chromium, mounted a volume with the media and it was much better. Not perfect, but better.

[nurdism]

oh if that's the case, that might be a limitation of running a browser in a docker container.

[stevelord]

It appears the pipeline is quite complex if you think about it, we have:

• A browser that has to decode video, render images, play sound and draw it all in X11
• A Pulseaudio setup that has to handle sound
• An X11VNC instance that has to take the images and convert to VNC (I think)
• A Gstreamer pipeline that has to reencode everything
• A Nodejs setup that then has to stream the output to multiple browsers

I think I'm wrong about some of the above, I've only seen the parts individually and haven't spent enough time digging to be sure what's used when. I wonder if there's a more optimal way of handling this. I'm going to try looking at the IO stats next time I run Neko and see if the content is being written to disk, read from disk repeatedly by different stages.

[nurdism]

You almost have it right, the current system is :

• Browser does its thing
• X11 via xserver-xorg-video-dummy
• Pulseaudio sinks
• Neko (using golang) uses Gstreamer pipelines and captures raw X11/Pulseaudio output and encodes, then sends them to webrtc clients.

The only improvements I can see is to optimize the Gstreamer pipelines or find a better webrtc lib/alternative lib, the client is 100% static html/js once its been generated so no performance impact there.

[zaeek]

Hello, can u rewrite the code in js/node , it's hard for me to understand the code in vue.js.
Thanks 🗡️

[stevelord]

@zaeek that might be best served as at least as a separate issue, if not also a pull request.

[stevelord]

As an aside, I ran another session last weekend and had horrible dropouts. It looks like some of this may be coming from latency on my broadband connection introduced through quarantine and furlough. I've noticed this isn't happening with Plex, but plex is streaming video, where as Neko will stream individual frame state.

[GigaFyde]

That is indeed caused by the differences in how both serve the video feed,