Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Error when Redis not ready #3284

Open
1 task done
kdambekalns opened this issue Jan 26, 2024 · 3 comments
Open
1 task done

BUG: Error when Redis not ready #3284

kdambekalns opened this issue Jan 26, 2024 · 3 comments
Assignees
Labels

Comments

@kdambekalns
Copy link
Member

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

In a project we are seing exceptions after a deployment. The cause is a failing doctrine proxy compilation step early on, that in turn is caused by the RedisBackend trying to read Redis, but that just says: LOADING Redis is loading the dataset in memory.

Expected Behavior

It should "Just Work" (tm) 🙃

Steps To Reproduce

No response

Environment

- Flow:
- PHP:

Anything else?

No response

@kdambekalns kdambekalns self-assigned this Jan 26, 2024
@kdambekalns
Copy link
Member Author

The RedisBackend could at least wait a tiny bit. Like this?

public function get(string $entryIdentifier)
{
    $tries = 1;
    do {
        try {
            $result = $this->redis->get($this->getPrefixedIdentifier('entry:' . $entryIdentifier));
            break;
        } catch (\RedisException  $exception) {
        } finally {
            if ($tries > 10) {
                throw $exception;
            }
            usleep($tries * 100);
            $tries++;
        }
    } while (true);

    return $this->uncompress($result);
}

@kitsunet
Copy link
Member

Totally could and should, two things come to mind: incrementally increase sleep time (maybe going from 1 up with ^2) and we would need to etst what happens in case of a full outtage, because if then multiple gets potentially wait before something crashes it could get long and lead to a self DDos kind of state?

And we might want to do that for set and other stuff as well?

Also IMHO this particular case going wrong should be fixed in deployment, the ready state of redis before deployment is "live" should be bound to the loading being finished I guess.

@kdambekalns
Copy link
Member Author

incrementally increase sleep time (maybe going from 1 up with ^2)

It does that, but you want it to be exponential? Yeah, why not.

And we might want to do that for set and other stuff as well?

Not sure we need that. The effect I see is that Flow tries to read something while still doing "compile stuff", so if that waits for Redis to become ready, anything after that would work.

I'd say writing before even attempting to read is an edge-case, no?

the ready state of redis before deployment is "live" should be bound to the loading being finished I guess.

That was my immediate reaction, but some Redis commands do work before Redis is done loading, so it is expected behaviour for it to accept connections.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants