Backblaze B2 "no tomes available" and related errors

[cadusilva]

Steps to reproduce the problem

1. Try to upload some media, like a picture
2. Wait until the app or web UI do their thing
3. The upload fails to complete
   ...

Expected behaviour

Upload successful

Actual behaviour

Upload failed

Detailed description

Users are reporting recurrent problems while uploading media. They can't finish the request due to issues with Backblaze. Maybe this is a particular behavior from this provider?

In Sidekiq, there are things like:

• Aws::S3::Errors::ServiceUnavailable: no tomes available
• Stoplight::Error::RedLight: object-storage
• Aws::S3::Errors::InternalError: An internal error occurred. Please retry your upload.

I tried to ask Backblaze about this "tomes" thing and their answer was the following:

The no tomes available message indicates that the particular connection you are attempting to reach our server with is
saturated and cannot accept any further connections. To get around this, your server should attempt to reconnect to the
servers, which will trigger the connection to retry with a different tomb that does not have as many incoming connections. 
 
We have an article regarding this here:
https://www.backblaze.com/blog/b2-503-500-server-error/

So my layman guess is that they have some peculiar way of work that results in these messages and failures to upload, when Mastodon saturates a particular connection instead of asking for a new, unsaturated one.

I also tried to increase S3_OPEN_TIMEOUT and S3_READ_TIMEOUT to 20 seconds, but didn't help [much or at all]. The issue isn't a constant, but a intermittent one.

Ursal is one of the biggest South American instances, with a vast federation and lots of media from users and from the Global timeline to deal with. But I guess this problem isn't common as there are no results after searching for "no tomes available".

I also have a smaller instance with way fewer users but also using Backblaze B2 and from time to time there is this upload issue with the same Sidekiq errors.

Mastodon instance

ursal.zone

Mastodon version

4.3.0-alpha.3-35b517c

Technical details

If this is happening on your own Mastodon server, please fill out those:

• Ruby version: 3.2.3
• Node.js version: v20.12.2

[markrsmith83]

I am having an identical problem on 4.2.8. Backblaze's answer is that the app should retry with increasing time between requests on receiving a 500 or 503.

[cadusilva]

I am having an identical problem on 4.2.8. Backblaze's answer is that the app should retry with increasing time between requests on receiving a 500 or 503.

Yes, this issue wasn't introduced with the alpha version, but doesn't seem to be a problem with Mastodon itself, instead it looks like something related with the inner workings of B2 service.

While there's no workaround in place, for instances with at least hundreds of active users or using big relays, I guess B2 is not a good fit anymore.

Earlier today I saw this, that's not exactly the same issue, but another evidence of the particular modus operandi from B2 service.