Implement Log Rotation in the kratos container

[bencekov]

Enhancement Proposal

State of charm:
Currently the following is how logs are being handled by the kratos charm:

• The kratos application writes its logs to standard error and standard output streams
• The streams are piped into tee linux utility program
• tee writes logs to standard out and /var/log/kratos.log
• because of stdout logs can be viewed with kubectl logs and juju debug-log
• the charm code downloads promtail binary and starts promtail service in the kratos container
• promtail pushes logs in /var/log/kratos.log to loki deployment (service is discovered through juju integration)

Problem:
There's no log rotation implemented in the charm. The size of the log file will grow, until k8s restarts the container for taking up too much resources. This results in downtime.

Things to consider:

• Ory products write logs to the standard streams. It's not configurable. Documentation recommends users pipe output to file.
• When used with static_scraping configuration, promtail doesn't support regex for finding the log files.
• The Kratos Rock doesn't have systemd or any syslog services running, therefor promtail's features involving streaming logs from systemd journal or any other service is not available
• As I understand, in a K8s environment promtail is deployed as a k8s daemon with privileged containers that read the files where containers standard streams are piped into.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/IAM-881.

This message was autogenerated