Accessing large files from public buckets scares the users

[adamnovak]

I had someone complain to me that Toil was broken because it logged:

[2024-04-09T13:05:31-0700] [MainThread] [W] [toil.lib.aws.utils] Strategy 1 to get bucket location did not work: An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
[2024-04-09T13:05:31-0700] [MainThread] [W] [toil.lib.aws.utils] Strategy 2 to get bucket location did not work: An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
[2024-04-09T13:05:33-0700] [MainThread] [E] [toil.lib.retry] Got a <class 'botocore.exceptions.ClientError'>: An error occurred (403) when calling the HeadBucket operation: Forbidden which is not retriable according to <function retryable_s3_errors at 0x7fed68f4a560>

And then it sat there for several minutes.

What actually happened was that it finally worked out how to access the data in the S3 bucket, but presumably it logged its successes at a lower level than its (often expected) failures. So to the user it looked like something went wrong.

We should maybe buffer all this information about the things that didn't work and not log it if we find a way that does work. Or we should announce at the same log level when we actually start downloading the thing.

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1539