Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Please implement back off for matrix sync retries #2016

Open
gaykitty opened this issue Feb 1, 2024 · 3 comments
Open

Please implement back off for matrix sync retries #2016

gaykitty opened this issue Feb 1, 2024 · 3 comments

Comments

@gaykitty
Copy link

gaykitty commented Feb 1, 2024

Description

I run a Matrix server. We had a couple days of down time last week, during which we were getting absolutely spammed by sync requests from a single user. Over 2 million in 24 hours. After some sleuthing I figured out it was an instance of this bot sending the requests.

Steps to Reproduce

Run the bot against a matrix server that returns error when sync is called.

Expected Functionality

Implement some kind of back off if sync fails to multiple times in a row.

Experienced Functionality

Because our Matrix server was down, but our Nginx server that sits in front of it was still up, all requests immediately returned HTTP 502. So, every call to sync the bot made immediately returned an error, and the bot immediately tried the request again.

while True: # pylint: disable=R1702
response = await self.connection.sync(
timeout=int(60 * 1e3), # 1m in ms
sync_filter=self.filter_id,
since=self.connection.sync_token,
)
if isinstance(response, nio.SyncError):
_LOGGER.error(
f"Error during sync: {response.message} (status code {response.status_code})"
)
continue

This loop should implement some kind of back off when multiple error occur in a row to prevent this.

@RaitoBezarius
Copy link
Contributor

I'm so sorry for that, I will implement exp back off ASAP and in the bot I run on your instance.

@gaykitty
Copy link
Author

gaykitty commented Feb 6, 2024

Thank you very much 💖

@Cadair
Copy link
Contributor

Cadair commented Feb 6, 2024

I would be very happy to review a PR for this, but wont have time to write one myself. Sorry this project has got a little quiet recently, things are very busy for all of us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants