ZMC exhausts memory and crashes #3772
Comments
|
I have the same problem. When a rtsp camera is not reachable zmc consumes all the memory on the server. This seems to happen during the restart process of zmc. This has been reported many times in the forum. Setting max buffer does nothing because the stream is never started. https://forums.zoneminder.com/viewtopic.php?t=32739 This is also reproducible in 1.37.x |
|
I am running into the same issue, I believe. After the server went down a few times due to OOM, I set up cgroups to limit memory consumption. It hit the mem caps, however it was still running. Here's what it looks like normally: This was on 1.37.45~20230926121048-focal from the ppa. |
|
Experiencing the same thing - currently running v1.37.43 |
|
@jrtaylor71 is correct. There is a direct correlation to the camera not being able to be accessed and the memory consumption. |
|
I have not been able to recreate this here. Are the FFmpeg type or Remote type or something else? I need a debug level 3 log from one of the zmc processes. |
|
@connortechnology ffmpeg in my case. I agree with @thisabstractmind that this might be related to camera not being accessible - i have 5 monitors accessed through vpn tunnel over internet, so their streams aren't as stable as local network and memory seems to climb after short network outages. I'm away atm, so can't get you the debug logs. |
|
The same issue. I'm using 4 cheap WiFi Yi camera with YiHack to add RTSP server. Cameras are cheap and with constrained resources so RTSP stream is not stable (parallel background stream is still running to manufacturer server). All cameras are added as I'm using unprivileged LXC container in Proxmox. System is Debian. Zone Minder was working correctly for months (with high CPU usage due to video analysis but stability was OK). I think problem begins after one of system update. Now I'm using latest Debian Bookworm but today I've added deb multimedia repo, so ffmpeg + libraries were updated. |
|
Debug from 3 cameras. Camera 5 is dead right now, camera's 7 and 12 I simulated a switch failure by rebooting it. All 3 camera's are are powered from an Avaya ERS5520 and run on vlan 555. Before test After test After a larger failure like this I have had to reboot the vm to release the memory. Restarting the zoneminder service does not help. zoneminder 1.36.33-jammy1 |
|
If restarting ZM doesn't free up the memory... then it's not ZM that is consuming it. ANyways, thos debug logs show zm connecting and disconnection to the offline camera.. unfortunately no information as to why memory would be used. In my own testing, I setup top, hit M to sort by memory use... and then power cycled a camera and watched what happened. What happened is that ZM had a read failure, shut down everything, freed up all ram, and then retried the connection as it should WHen the connection came back it resumed and didn't use any excess ram. What you logs do NOT show, that I expected to see, was a resource shortage. THey seem to be happy. So I'm stumped. |
|
I have always tested with more than one camera failure and when it has happened it has always been more than one. In fact the more cameras that go offline the faster the memory gets used up. As for freeing the memory I should have pointed out that cache and swap are never free'd but I have seen zm crash like the above output without a reboot. Just a service restart free's the acvtive memory. I have included some screen shots from munin that I have had running to track usage. |
|
Latest build has a fix for a small memleak when trying to reconnect to a dead camera. So might fix this. Shouldn't leak too quickly though, so maybe not. |
|
On what version? I think most of us that reported this are still on 1.36. |
|
I updated last week and tried it again, but I still seem to be having an issue with the memory use growing significantly. I let it go for a while and generally it stays pretty consistent in memory usage, but sometimes it starts consuming more than the normal amount of memory. It gets oom-killed by the cgroup limit before it causes problems for the rest of the machine. I do have multiple cameras in use, and the memory in one may increase significantly while the others are still fine. Due to backups, sometimes the CPU load on the machine is high, and other times it's nearly Zoneminder alone using the cpu cycles for mocap. They are all ffmpeg streams, and I have tried both tcp and udp rtsp streams with no significant difference. I set up a script (taking 1 sample per min) to record the memory usage over time, along with the absolute time (to use in correlation with the logs) and the process time (in seconds) and the rss/vsz to show when it starts going up. Here's an excerpt of the memory usage log where it nears the end, right before it gets killed: The memory log and debug log are attached, along with the crash data. I trimmed the debug log to within the last ~11.5 minutes so it wouldn't be too large. I do have several full logs along with corresponding memory usage logs if I should send anything else. This is on 1.37.50~20240202093203-focal - I updated to current shortly before starting tests. I am not sure, but it's definitely possible there may be more than one issue at play as I haven't noticed significant increases in memory usage when I do try disconnecting/rebooting the cameras, and I am not on 1.36 like some of the others. |
|
@Middim Thanks for the quality info. So at first glance, I see everything falling behind around 5:11. I see a ton of warnings from ffmpeg which could be bad network, but tend to more often be that your cpu is not keeping up. I also see that you are saving jpegs. Encoding jpegs is surprisingly cpu intensive. If you want the analysis images, then only save those. Saving jpegs is not feasible past 720p. Certainly not at 20fps. |
|
@connortechnology I am the OP. I have not had an issue since changing video writer to pass through and disabling "save JPEGs". I think it was a bottleneck problem that led to delay and extra memory consumption. This was about 3 months ago. |
|
Yup, that's what I think this issue is. Saving jpegs is just not feasible at today's resolutions. We could use hwaccel for it, but why bother when we could encode an mp4. Long will likely get rid of the analysis jpegs as well in favour of storing an svg of the info and just overlaying that. |
|
I'm not saving jpegs, but i am saving audio and decoding keyframes+ondemand. I upgraded to 1.37.50 a few days ago and swap isn't growing a fast, but still seems to be growing. Need to give it a few more days though. |
|
Thanks for the info! I changed it so it only saves analysis image - and an SVG option for the analysis info sounds great too! It is likely that there would be cpu slowdown around that time as I have some backup processes that run through the night. In the past I have had the problem occur at random times. It would even occasionally retain large amounts of memory all day - but the recent changes that were made may already have resolved that as I didn't see that part occur again in the recent testing. I am retesting and will let you know what the results show after I collect some data in the new configuration. I hadn't considered the jpegs would cause a problem, but that would make an easy solution, and I am not attached to having them. One note though is that I have been saving things in this manner for a few years, but only had the problem since I updated from the older version of Zoneminder I was using last year, so I had attributed a change in process to something related to that. After that I also started using passthrough with the hopes that it would resolve some of the issue. While it didn't resolve the issue, it did significantly reduce the cpu load on the system. I'll post back if this configuration change fixes it, but it may take a few days if things are successful, due to the random nature of it occurring. Thanks again! |
|
@connortechnology thanks heaps for the fix! 1.37.50 seems to have resolved the memory leak i was experiencing. Swap is no longer filling up 🙏 |
|
@connortechnology Thanks for your help and for everything you do with this! My system is in a running state now. Just updating here as well, after running several days without hitting the memory limit. After setting analysis only jpegs, it has yet to crash, and since the patch it doesn't appear that the memory slowly grows in one direction. I have attached the memory logs of the two cameras I have running with data collection. Thanks!! |
|
I upgraded my system to 1.37.51 (current master) and there is an improvement with this issue. If a camera goes offline or is offline at startup the system no longer consumes all memory. The one thing I did find is that if you save a camera multiple times that is having a problem connect to an rtsp stream it runs the cpu usage up along with memory. A service restart cleared the cpu and memory usage. This happened to a camera that had a wrong password and I forgot what it was. |
|
Upgraded to 1.37.55~20240304175439-jammy a few days ago and I lost a wifi bridge link to a remote building. This caused zmc to crash on the server. It did recovery with no intervention but system is still slower than normal. Going to only restart zoneminder.
|
|
I've been having the same issue on 1.36.33 running on CentOS9 and Fedora CoreOS 40. The system is 4x 1080p15 cameras in to 8 monitors. System is a Core i7-8700T with 8gigs of RAM. Adding a swap files seems to have little effect (maybe give a few more hours before crash.) Adding RAM seems to have no effect either.
I get debug level 5 from the _zmc processes. I also have the logs from the other processes running at debug level. The full zmc log is 6gigs, I can send it out-of-band if needed. |
|
Your 150 pre-frame count means you will be holding 153 images in memory. At 1080p thats a lot of ram. 8Gb is really not enough. |
|
I increased the RAM to 16Gig and there is no discernable difference in performance. Zoneminder is still killed by zoneminder_crash-20240510-1146.log The system during normal operation seems not overly taxed. |
|
The system ran steadily for 15 minutes, then jumped to 8 gigs at about 7 minutes in. Then jumped to 16gigs, pegged all the processors and was killed. This behavior happens on all the systems we have. zoneminder_crash-20240510-1204.log
|
|
@connortechnology This problem seems to be getting worse again. I have 2 systems running current master branch and zmc seems to be crashing again. One of the systems required a reboot. |
Describe Your Environment
ZM version : 1.36.33
Installed by isaac ppa
Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-84-generic x86_64)
zmc process has a possible memory leak, start the zoneminder service and will eventually run out of memory, killing the process. The process will start over again and this will repeat.
Expected behavior
To run without exploding.
Debug Logs
The text was updated successfully, but these errors were encountered: