Add cluster crash recovery itest

[sed-i]

Enhancement Proposal

Would be handy to have a cluster crash recovery itest, for when, for example, daemon-kubelite fails to (re)start.

After manually restarting daemon-kubelite, the deployment was stuck on unknown/lost.

Model               Controller  Cloud/Region        Version  SLA          Timestamp
cos-lite-load-test  uk8s        microk8s/localhost  2.9.34   unsupported  19:43:43Z

App            Version  Status   Scale  Charm                         Channel  Rev  Address         Exposed  Message
alertmanager   0.23.0   active       1  alertmanager-k8s              edge      37  10.152.183.156  no       
catalogue               active       1  catalogue-k8s                 edge       5  10.152.183.232  no       
cos-config     3.5.0    active       1  cos-configuration-k8s         edge      14  10.152.183.20   no       
grafana        9.2.1    waiting    0/1  grafana-k8s                   edge      55  10.152.183.105  no       waiting for units to settle down
loki           2.4.1    waiting    0/1  loki-k8s                      edge      47  10.152.183.135  no       waiting for units to settle down
prometheus     2.33.5   waiting    0/1  prometheus-k8s                edge      87  10.152.183.141  no       waiting for units to settle down
scrape-config  n/a      active       1  prometheus-scrape-config-k8s  edge      38  10.152.183.231  no       
scrape-target  n/a      active       1  prometheus-scrape-target-k8s  edge      23  10.152.183.233  no       
traefik                 waiting    0/1  traefik-k8s                   edge      95  10.128.0.6      no       waiting for units to settle down

Unit              Workload  Agent  Address      Ports  Message
alertmanager/0*   active    idle   10.1.79.238         
catalogue/0*      active    idle   10.1.79.215         
cos-config/0*     active    idle   10.1.79.197         
grafana/0         unknown   lost                       agent lost, see 'juju show-status-log grafana/0'
loki/0            unknown   lost                       agent lost, see 'juju show-status-log loki/0'
prometheus/0      unknown   lost                       agent lost, see 'juju show-status-log prometheus/0'
scrape-config/0*  active    idle   10.1.79.204         
scrape-target/0*  active    idle   10.1.79.213         
traefik/0         unknown   lost                       agent lost, see 'juju show-status-log traefik/0'

[simskij]

duplicate