You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We're hitting lots of "503 The service is currently unavailable" errors in the Google AutoML warmup process.
With the increase in the number of AutoML locales the time to do the actual warmup also increases. We are at ~20 locales and it almost always takes more than the warmup job frequency (1 minute) to warm them all up.
Since we can only run one task in parallel, we no longer warmup locales every minute. The longer the time between the warmups, the higher the chance of hitting the cold engine, manifested as "503 The service is currently unavailable".
We use APScheduler for that task, so the simplest trick to test is to increase the number of instances that can run at the same time.
The text was updated successfully, but these errors were encountered:
We're hitting lots of "503 The service is currently unavailable" errors in the Google AutoML warmup process.
With the increase in the number of AutoML locales the time to do the actual warmup also increases. We are at ~20 locales and it almost always takes more than the warmup job frequency (1 minute) to warm them all up.
Since we can only run one task in parallel, we no longer warmup locales every minute. The longer the time between the warmups, the higher the chance of hitting the cold engine, manifested as "503 The service is currently unavailable".
We use APScheduler for that task, so the simplest trick to test is to increase the number of instances that can run at the same time.
The text was updated successfully, but these errors were encountered: