Could you include the rate-limit that you've configured your AWS S3 bucket?

[Jonathan-Gore]

Currently there is no documentation on when a rate-limit will be kicked in for downloading data from the iNaturalist. I arbitrarily chose 1/image a second and that almost instantly hit the rate-limit.

[alexshepard]

if you're downloading photos from the open data bucket then I don't believe there is a rate limit

[Jonathan-Gore]

Weird, I was definitely hitting a throttle.

I can try to hit it again and post the HTTP request error message here if you would like. But it sounds like this isn't something in your direct control.

Arbitrarily after downloading "too many" images I found 1 image every 3 seconds satisfied whatever throttle I was hitting. Doesn't help I got impatient and switched to multi-threaded for parallel S3 bucket access :p.

And to be clear, this wasn't an "image downloading slower" type of throttle, it was a full http request rejection.