Google Arts and Culture - Uncaught Error 429 Too Many Requests

[surao101]

URL
https://artsandculture.google.com/asset/shipwreck-on-the-coast-eug%C3%A8ne-delacroix/7QEvX2VgBvI66Q

Dezoomify option
Google Arts & Culture

Error
Uncaught Error: Unable to fetch https://artsandculture.google.com/asset/shipwreck-on-the-coast-eug%C3%A8ne-delacroix/7QEvX2VgBvI66Q
The server responded:
fopen(https://artsandculture.google.com/asset/shipwreck-on-the-coast-eug%C3%A8ne-delacroix/7QEvX2VgBvI66Q): failed to open stream: HTTP request failed! HTTP/1.0 429 Too Many Requests

(https://ophir.alwaysdata.net/dezoomify/zoommanager.js:394)

Browser used
Chrome Version 78.0.3904.70 (Official Build) (64-bit)

Solution

See latest answer: #337 (comment)

[lovasoa]

I see the same error. It looks like too many people have been using the dezoomify server at the same time, so google blacklisted the IP of the server. I guess the block is temporary, so you can try again later.
If it is not, then you can simply run dezoomify locally.

[surao101]

How can I run it locally?

[lovasoa]

I think the easiest to run locally is dezoomify-rs: https://github.com/lovasoa/dezoomify-rs

[surao101]

Thanks

[lovasoa]

let's leave that open to make it more visible and to track the issue

[lovasoa]

It looks like the ban on our IP has expired. However, this is a good reminder that I should implement some kind of rate limit so that it doesn't happen again.

[surao101]

Thanks, I checked as well, it works now.

[lovasoa]

Problem description

I set up a cloudflare worker to work around this issue. Every day, the first 100 000 requests are served through clouflare's servers (and not dezoomify's own server). Since cloudflare has many servers with many IPs all over the world, you are unlikely to get a Too many requests error if you are in these 100 000 first requests. After that, your requests go through dezoomify's server, and if many users load google images at the same time, you will see an error.

So if you encounter this error, the solution is simply to try again the next morning (the limit resets at 00:00 UTC every day).

Cloudflare also offers a paid plan at 5$ for the first 10 million requests + 0.5$ / 1 million requests. If someone is ready to make a donation of 5 or 6$/month, I can switch to that plan.

[lovasoa]

Update Our main host is now also limiting the number of requests we can make every day, meaning you are probably going to see this message more often.

You can donate here to participate to the hosting expenses: https://github.com/sponsors/lovasoa
Just 5 people donating $1 per month would be enough to upgrade our cloudflare subscription.

[lovasoa]

Good news ! We've reached $10 monthly donations, and I bought the basic cloudflare worker plan for dezoomify. I'll keep you updated if there are any issues, but you should already be able to use dezoomify normally without seeing any "error 429" now. Let me know if this is not the case.

If the donations keep growing that fast, I'll add an option to dezoomify to allow querying all image tiles through our proxy. This would generate much more requests and increase the cloudflare bill, but would allow downloading images from web servers that try to block dezoomify with a referer check.