WebRTC stream stops working

[mnorrsken]

What happened

Installed the latest version with all the compiling etc, got it running with WebRTC it was very nice for a while, but after a few minutes/hours it just stops working. /stream and /snapshot still works but not /webrtc

It starts working again if I restart crowsnest service from within mainsail.

Ive reverted and disabled RTSP and only running MJPG again. Not a big deal, I just wanted to report this.

What did you expect to happen

WebRTC streaming working indefinitely

How to reproduce

This is on my MainsailOS 1.0.1 RPi 3 b+ fitted with a Logitech C270 USB camera. Reproduced by running WebRTC for a while in Mainsail (with a web client open for approx an hour or so).

Additional information

I didn't find any errors, and cpu/mem usage of host was ok, but I found a few suspect websocket lines in journal.
crowsnest.txt
suspect.txt

[mnorrsken]

Woops this seems to be a camera-streamer bug. Will test some things and report back here before closing.

[mnorrsken]

I increased my RPi video ram from 160 to 256mb and this particular issue seems to be gone, or at least havnt happened for 12+ hours.

[PSandro]

increasing the GPU memory also resolved this issue for me. Should we take this up into documentation/readme?

[mdvorak]

Increasing video ram did not fix the issue for me. It still fails after a while, streaming 800x600 resolution from USB webcam.

[stonemandave]

Still an issue. I have 2 printers same setup and one works while the other just stops.

[4rianton]

still an issue..my rtc feed keeps stopping with both of my cameras. GPU memory set to 256..

[SaltyPaws]

I can confirm I have the same issue on both my Voron zero and 2.4 mainly with raspicam v1. The voron 2.4 is also running a logitech USB webcam, which keeps running significantly longer compared to the raspicam but eventually also goes black. a reboot solves the issue.
hardware:
2.4
/dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_26E11DCF-
Raspberry Pi 4 Model B Rev 1.1

both systems:
Operating System: Raspbian GNU/Linux 11 (bullseye)
Kernel: Linux 6.1.47-v8+
Architecture: arm64

Raspicam v1: /base/soc/i2c0mux/i2c@1/ov5647@36
Computer: RPI4
Software version (latest as of oct 25)
crowsnest v4.0.4-6-g767c53aa
fluidd v1.26.1
klipper v0.11.0-299-gb1f597c5
Klipper-Adaptive-Meshing-Purging v1.1.2-4-g2389a994
moonraker v0.8.0-186-g2641fc54

gpumem: 128

[SaltyPaws]

Even with gpumem 256, the WebRTC for the raspicam still stops streaming.

[mryel00]

Forgot to reopen this issue. Can you guys please try to use --camera-force_active=1 as a custom_flag?

[recrudesce]

Forgot to reopen this issue. Can you guys please try to use --camera-force_active=1 as a custom_flag?

Where do we set this ?
Never mind, worked it out :)

[mryel00]

Where do we set this ?

Inside the crowsnest.conf under custom_flags

[recrudesce]

Literally just worked it out as you replied :P Thanks !

[mryel00]

Also just as an overall information for this. We couldn't reproduce this. So please attach a debug log next time with the issue reproduced, that we can maybe try to make some assumptions based on the output.

[SaltyPaws]

crowsnest.log

With --camera-force_active=1 flag in combination with gpumem=256 I have had no more issues. After 48hr+ both cameras are still running. I have attached my log file. Perhaps we just need to update the documentation. I will report back if I encounter any more issues.

[mryel00]

@SaltyPaws thank you for the confirmation, that this works for you too. Next time a debug log would be nice. To create one, just set log_level: debug inside your config. Someone on the Discord found out that this flag might help and there was a line about restarting the camera inside the debug log. So if something like that also appears for others, it's most likely the cam itself that shuts down.
Also I would need some confirmation, if the gpumem=256 is needed too or if --camera-force_active=1 is enough. In the case of the latter, I would just add it to the default start parameters.

[SaltyPaws]

To further confirm this is the right fix, after 5 days, my USB camera shutdown, it did not have --camera-force_active=1 in the config. The Raspicam V1 was still up and running (it has the flag). Wow, wait 5 days to confirm a bug, I would call that hard to find! I will update the flag for both cams, and change gpumem back to 128 to test.

[SaltyPaws]

@SaltyPaws thank you for the confirmation, that this works for you too. Next time a debug log would be nice. To create one, just set log_level: debug inside your config. Someone on the Discord found out that this flag might help and there was a line about restarting the camera inside the debug log. So if something like that also appears for others, it's most likely the cam itself that shuts down. Also I would need some confirmation, if the gpumem=256 is needed too or if --camera-force_active=1 is enough. In the case of the latter, I would just add it to the default start parameters.

Looks like gpumem=256 is not needed. Running for 2 days now with gpumem=128 and --camera-force_active=1 on both cameras, everything is still up and running.

[mryel00]

@SaltyPaws Just for your information. --camera-force_active=1 is now set by default, since Crowsnest v4.0.5 released 2 days ago.

[mryel00]

We had to remove this parameter from default values as it causes problems with some printers #202
So please set --camera-force_active=1 again manually inside the custom_flags. We are already in touch with the developer of camera-streamer to find a solution for this.

v4.0.5 has it set by default and v4.1.0 removed that default again.
v4.1.0 only added one new feature #203 that is most likely only needed for SpeederPads and changed some things on the installer. So you can stay on v4.0.5 for now, if you like, without missing out on cool new features.

[Gyrohammer]

I've just had this happen as well with a generic USB cam. Logs indicate no issue, crowsnest looks normal. A restart fixes it but it goes offline within three to four days.

I just restarted the Pi, crowsnest has been updated to v4.1.0 and --camera-force_active=1 has been applied. I'll update this when it goes offline next to get some kind of timeline.

Edit: The camera has just gone offline again, I've restarted crowsnest and its blank. Unsure what happened, crowsnest logs look fine. Going to try this solution and see how long it holds.

[mryel00]

Edit: The camera has just gone offline again, I've restarted crowsnest and its blank. Unsure what happened, crowsnest logs look fine. Going to try this solution and see how long it holds.

This shouldn't help in any way for this. It seems like there was a bug with ustreamer in the beginning, that one is already fixed for quite some time. Also this problem is about camera-streamer and not ustreamer.

Within 3-4 days I would consider fine. camera-streamer WebRTC isn't perfect, but this issue is something that is pretty hard to debug as it seems like appearing at random. --camera-force_active=1 seems to help but there are still some user reporting issues with a "fast" disconnect, like you for example.

I also recommend everyone to always attach a log as there are some good and maybe helpful informations about your setup.

[hardKOrr]

I'm running into this issue as well, I have a debug
crowsnest(12).log.

I am on a Raspberry Pi 3b+ using pi camera v1.2, I've set gpu memory to 256 and also added --camera-force_active=1 the pi is on WiFi.

[recrudesce]

I get this issue with RTSP streams too - after a while they just stop working.
Pi4, MainsailOS, USB camera.

Usually restarting crowsnest fixes the issue, unless it kernel panics the Pi, of course

crowsnest.log

Log attached.

[mryel00]

@recrudesce Ofc the rtsp will break down too. Both are provided by camera-streamer.
You don't have uploaded your log yet?
Does this kernel panic happens as soon as it breaks down or is it related to something else? Do you use the --camera-force_active=1 parameter?

[recrudesce]

log uploaded, no idea why github didn't like it before.

The kernel panic happens sometimes if I restart the service after it stops working. Not all the time, but sometimes. Then the SSH connection dies and the only way I can get back into the Pi is to physically unplug it to power it down.

I do use the parameter, yeah.

[mryel00]

I recommend to remove that parameter again. Your camera-streamer is restarting every 20s.
So you are one of the unlucky ones where this doesn't help.

[recrudesce]

I think this is a camera-streamer issue, because running camera-streamer separately (without crowsnest), I still get the crashes on service restart, and webrtc just stopping. I'll raise an issue over there.

[mryel00]

Crowsnest is just starting camera-streamer for you with maybe some extra parameters, like we tried with --camera-force_active=1.
I thought I already posted it here, but it doesn't seem like it. There is already a post about this on the camera-streamer GitHub: ayufan/camera-streamer#94

[meteyou]

@g14nluc4 please do not write your problem in any issue. If you need help/support, you are generally in the wrong place in the GitHub issue tracker. there is more information here: https://docs.mainsail.xyz/faq/getting-help