Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Adds prometheus alert rules #364

Merged
merged 5 commits into from
May 24, 2024
Merged

Conversation

saltiyazan
Copy link
Contributor

@saltiyazan saltiyazan commented May 8, 2024

Description

Adds Prometheus and Loki alert rules following spec TE071.

Initially this PR included Loki rules, but because of this bug those rules have been removed and should be addressed when the bug is fixed.

To test deploy the charm from this branch and follow the guide on how to integrate with COS, go to the alerts tabs in Grafana and then:

  • To test the vault sealed rule, seal and unseal vault and observe the alert firing
  • To test the leader last contact rule, reduce the threshold to something very small, maybe 5 ms and observe the alert firing
  • To test the failure tolerance rule scale vault up to 3 units then back to 2 units and observe the alert firing
  • I couldn't find a way to simulate leadership setup failure

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have added tests that validate the behaviour of the software
  • I validated that new and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

src/prometheus_alert_rules/vault_sealed.rule Outdated Show resolved Hide resolved
src/charm.py Outdated Show resolved Hide resolved
@saltiyazan saltiyazan marked this pull request as ready for review May 23, 2024 09:38
@saltiyazan saltiyazan requested a review from a team as a code owner May 23, 2024 09:38
@saltiyazan saltiyazan requested a review from gruyaume May 23, 2024 09:40
@saltiyazan saltiyazan changed the title feat: Adds prometheus and Loki alert rules feat: Adds prometheus alert rules May 23, 2024
Copy link
Contributor

@ghislainbourgeois ghislainbourgeois left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Your branch is out-of-date, please rebase or merge main on it before merging.

@saltiyazan saltiyazan enabled auto-merge (squash) May 24, 2024 12:31
@saltiyazan saltiyazan merged commit 34f436c into main May 24, 2024
13 checks passed
@saltiyazan saltiyazan deleted the TLSENG-198-add-alert-rules branch May 24, 2024 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants