New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory usage increases everytime tls.reload is executed #3823
Comments
I just realized that i forgot to mention.. in addition to the logged error message our clients start to get connection issues as well, so we have to restart Kamailio asap in that case.. |
@denzs do you have a monitoring tool? Prometheus + Graphana graphs? |
Probably this part has to be reviewed ... first the tls reload was initially designed to be done rather rarely, when the certificates expires. The CRL feature was also not much in use, at least in what I could experience so far, most of the deployments are with server-side only certificates. Furthermore, I am not sure if old certificates can be cleared right away after the restart, existing connections are not closed and there might be some references to their certificates. Are you doing the reload only if there are changes in the content of the crl or certificate files? Or the reload is done anyhow? |
@sergey-safarov yes we do :) @miconda at the moment we do the tls.reload unconditionally and quite 'high frequently' to ensure the CRLs are up to date.. of course we can check if the CRL changed, but from my point of view that would only delay the neccesary restart of kamailio.. |
The problem actually occured after we added the CRL some weeks ago.. without CRL there was no such behaviour. Anyhow i thought raising an issue makes sense, because from my point of view there is definitively some memory leaking when using tls.reload in combination with a CRL.. |
If it happens only with adding a CRL, it looks indeed like an issue in this code path. In the end using CRL is probably quite rare. |
After some time debuging, I could replicate this issue of memory increase when using a CRL and tls.reload. One possible issue according to memory statistics printed frequently while we have
Memory here increases until we exhaust the shared memory max allocation and then tls.reload fails. Some notes:
|
Description
We are using Kamailio 5.7.4 on Debian 12 (from http://deb.kamailio.org/kamailio57) with rtpengine as an Edgeproxy for our clients. The instance terminates SIP/TLS (with Cliencertificates) and forwards the SIP Traffic to internal systems.
After some days we are getting errors like this
tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: shm=7318616 threshold1=8912896
First we thought Kamailio just doesnt have enough memory, so we doubled it..
But after some days the Logmessage (and Userissues) occured again.
So we monitored the shmmem statistics and found that used and max_used are constantly growing til it reaches the limit.
As i mentioned we are using client-certificates and so we are also using the CRL feature.
We do have a systemd-timer which fetches the CRL every hour and runs 'kamcmd tls.reload' when finished.
Our tls.cfg looks like this:
After testing a bit we found that every time tls.reload is executed Kamailio consumes a bit more memory which eventually leads to all the memory being consumed which leads to issues for our users.
See following example:
Troubleshooting
Reproduction
Everytime tls.reload is called the memory consumptions grows..
Debugging Data
Log Messages
SIP Traffic
SIP doesnt seem to be relevant here
Possible Solutions
Calling tls.reload less often or restart kamailio before memory is consumed ;)
Additional Information
The text was updated successfully, but these errors were encountered: