feat: curio: alertManager

[LexLuthr]

Related Issues

Proposed Changes

This PRs adds a new singleton task for alerting.
The alerting task can create an incident for a previously configured service.
User should get critical alerts via SMS, email and call.

• Test in a running Curio cluster before merging

Additional Info

A new page about alerting is required in Curio docs. We also need to create a section on how to sign up and configure PagerDuty.

Checklist

Before you mark the PR ready for review, please make sure that:

• Commits have a clear commit message.
• PR title is in the form of of <PR type>: <area>: <change being made>
  • example: fix: mempool: Introduce a cache for valid signatures
  • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
  • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
• If the PR affects users (e.g., new feature, bug fix, system requirements change), update the CHANGELOG.md and add details to the UNRELEASED section.
• New features have usage guidelines and / or documentation updates in
  • Lotus Documentation
  • Discussion Tutorials
• Tests exist for new functionality or change in behavior
• CI is green

[magik6k]

Looks really good. Just one nit.

Not in this PR, but would be neat to reuse the config parsing logic in balanceCheck and gather all miner-ids served by the cluster, then check that winningPoSt is happenning correctly based on entries being added to the mining_tasks table (we expect ~2880 entries in last 24h per minerID based on the base_compute_time column, the check should probably be for a smaller window tho)

[LexLuthr]

It's alive!!!

Pushing the suggested changes and some minor changes in another commit.

[LexLuthr]

Looks really good. Just one nit.

Not in this PR, but would be neat to reuse the config parsing logic in balanceCheck and gather all miner-ids served by the cluster, then check that winningPoSt is happenning correctly based on entries being added to the mining_tasks table (we expect ~2880 entries in last 24h per minerID based on the base_compute_time column, the check should probably be for a smaller window tho)

I think a better check would be to just check that all "win" instances resulted in a block which was accepted by the chain. This would help see if SPs are loosing blocks due to any issues.

I have extracted the address function for future use.

[magik6k]

I think a better check would be to just check that all "win" instances resulted in a block which was accepted by the chain. This would help see if SPs are loosing blocks due to any issues.

That's another good one, but the former does tell you that you have some machines in the cluster taking care of winningPoSt at all times