Entire lockup of Guardian system, running locally

[mattsmithies]

Problem description

Having started to run the Guardian yesterday locally by upgrading to the latest version 2.23.1, I have been using it throughout today to create a new client against.

Specifically, I have simply been working with the login, register, token, and session endpoints before I start building out more advanced flows.

After using the API of guardian throughout the day it froze up completely, previously, before this time, I was testing the register flows.

Step to reproduce

Reproducibility in this case will be tricky, but I have run a couple of commands, including:

docker compose down
docker compose -d up 

Running docker ps yields this

CONTAINER ID   IMAGE                                  COMMAND                  CREATED          STATUS                    PORTS                                                                                        NAMES
98f4298c7ffb   hedera-guardian-web-proxy              "/docker-entrypoint.…"   13 minutes ago   Up 13 minutes             0.0.0.0:3000->80/tcp                                                                         hedera-guardian-web-proxy-1
878809b6661b   hedera-guardian-api-gateway            "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes             3002/tcp, 6555/tcp                                                                           hedera-guardian-api-gateway-1
bc5a50efae6a   hedera-guardian-application-events     "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes             0.0.0.0:3012->3012/tcp                                                                       hedera-guardian-application-events-1
7d1be65e7575   hedera-guardian-guardian-service       "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes             0.0.0.0:5007->5007/tcp, 6555/tcp                                                             hedera-guardian-guardian-service-1
aca8d17e50ed   hedera-guardian-policy-service         "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes             0/tcp, 0.0.0.0:5006->5006/tcp                                                                hedera-guardian-policy-service-1
6e46385684f2   hedera-guardian-worker-service         "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes             6555/tcp                                                                                     hedera-guardian-worker-service-2
0672d5e935ed   hedera-guardian-worker-service         "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes             6555/tcp                                                                                     hedera-guardian-worker-service-1
908ecddee2a0   hedera-guardian-auth-service           "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes             0.0.0.0:5005->5005/tcp, 6555/tcp                                                             hedera-guardian-auth-service-1
d3d411240b71   hedera-guardian-notification-service   "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes                                                                                                          hedera-guardian-notification-service-1
a481402a9b6f   hedera-guardian-logger-service         "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes             6555/tcp                                                                                     hedera-guardian-logger-service-1
0d307384c4ac   mongo-express:1.0.2-20                 "/sbin/tini -- /dock…"   13 minutes ago   Up 13 minutes             8081/tcp                                                                                     hedera-guardian-mongo-express-1
b8d87a862b6c   ipfs/kubo:v0.26.0                      "/sbin/tini -- /usr/…"   13 minutes ago   Up 13 minutes (healthy)   0.0.0.0:4001->4001/tcp, 0.0.0.0:5001->5001/tcp, 4001/udp, 0.0.0.0:8080->8080/tcp, 8081/tcp   hedera-guardian-ipfs-node-1
59496daff47c   nats:2.9.24                            "/nats-server --http…"   13 minutes ago   Up 13 minutes             4222/tcp, 6222/tcp, 0.0.0.0:8222->8222/tcp                                                   hedera-guardian-message-broker-1
3ab891ae2180   prom/prometheus:v2.44.0                "/bin/prometheus --c…"   13 minutes ago   Up 13 minutes             0.0.0.0:9090->9090/tcp                                                                       hedera-guardian-prometheus-1
cf7285271fa1   mongo:6.0.13                           "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes             27017/tcp                                                                                    hedera-guardian-mongo-1
52f4e7100776   hedera-guardian-topic-viewer           "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes             3006/tcp, 0.0.0.0:5009->5009/tcp                                                             hedera-guardian-topic-viewer-1
b9fd0330a2b1   hashicorp/vault:1.12.11                "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes             0.0.0.0:8200->8200/tcp                                                                       hedera-guardian-vault-1
66213f53878f   grafana/grafana:10.0.10                "/run.sh"                13 minutes ago   Up 13 minutes             3000/tcp, 0.0.0.0:9080->9080/tcp                                                             hedera-guardian-grafana-1
def27ca4aef0   hedera-guardian-mrv-sender             "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes             3003/tcp, 0.0.0.0:5008->5008/tcp                                                             hedera-guardian-mrv-sender-1

And as a point of confirmation, the API stopped working.

Expected behavior

Just for the Guardian to run and work accordingly.

The question would be, does any of the authentication methods change state or have side effects?

Screenshots

[mattsmithies]

I tried using the system this morning, just logging in and while I could see the login screen, I received this error notification on the bottom right of the screen.

In addition, the entire API seems like it's down. The question would be, is this an instance of the database becoming corrupt?

[mattsmithies]

I am running on macOS 13.4.1, but I have managed to successfully build and run the software initially with test API calls to some basic authentication endpoints.

Using a "develop" env file, in conjunction with docker compose up -d --build

And, before this upgrade, I ran the troubleshoot directions:

docker builder prune --all
docker compose build --no-cache

[mattsmithies]

Okay, so I continued digging. And I found out that my standard registry user was deleted.

Effectively, I have a test that checks whether a user cannot be registered -- so a standard registry user, we use the name 'dovuauthority'.

If you look at the screenshot, I was able to generate a duplicate user, so maybe data was lost.

This below image is my test that I run to check if Registration of a user of a role of registry.

[mattsmithies]

So, I logged in with the new standard registry user, and it's freezing up on the initial registration screen for setting up your keys.

Has anyone else experienced this behaviour?

[mattsmithies]

Okay, another update.

I believe I've resolved the issue (but not the apparent data loss that was suffered), I will continue on my R&D quest.

I am able to login as a user and run my tests again, it feels inconsistent, but if I hit this issue again, I will add more comments.

[mattsmithies]

Okay, I believe I think I understand what is happening.

I've had complete DB data loss three times now in the last week on my local machine.

This tends to happen when a task is unable to complete as it has reached a state whereby it is unable to finish -- and almost has an "infinite loop" effect system-wide.

Perhaps the data is in an invalid state for logic to continue?

What I have observed, is that while there are timeout functions for tasks, it almost attempts to create new tasks on top (or loops), and then blocks up the entire task queue and underlying event loop.

This has the downstream effect that folks are unable to view data through the Guardian frontend as the ability to fetch data from the API has been completely blocked.

[mattsmithies]

This could be related to #3520 (only time will tell)