Prometheus stuck in installing charm software

[kaskavel]

Bug Description

Solutions QA team has a failed run when deploying COS layer (latest/stable:11) on top of microk8s v.1.29.0, prometheus charm stayed in maintenance state with the message: "installing charm software", until it timed out after

From the logs, no indication of the root cause of the error was found.

Failed run:

https://solutions.qa.canonical.com/testruns/25f56cba-e9c3-4ca2-abd5-c807b6d8b902

Logs:

https://oil-jenkins.canonical.com/artifacts/25f56cba-e9c3-4ca2-abd5-c807b6d8b902/index.html

To Reproduce

On top of MAAS 3.3.5, we bootsrtap a juju 3.3.1 controller, we deploy microk8s v.1.29 and the COS on latest/stable.

Environment

This test was ran on virtual machines on our lab3-litleo

Relevant log output

prometheus/0     maintenance  idle                        installing charm software

Additional context

No response

[IbraAoad]

Hey @kaskavel ,

Would you be able to re-run this?, as we're trying to reproduce this on stable but can't.

[motjuste]

We have another occurrence of this bug in a similar deployment in Solutions-QA labs, although with different versions of MAAS (3.4.0), Juju (2.9.46), and microk8s (1.28.7).