have https protocol driver automatically back off when it gets the Too Many Requests error?

[petersilva]

so... sr3 processes messages in batches, and amortizes connections over those batches. If one message transfer fails, it just tries the next one. For many cases (like 404) that's fine, maybe the next one does exist, but in the Too Many Requests case, it will never help, and likely hurt:

024-02-13 22:30:03,497 [ERROR] sarracenia.flow do_download gave up downloading for now, appending to retry queue
2024-02-13 22:30:03,548 [ERROR] sarracenia.transfer.https __open__ Download failed 4 https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e03d0f4a073c4bbdee/data?time-series-code=wlo&from=2024-02-13T18:29:50Z&to=2024-02-13T20:29:50Z
2024-02-13 22:30:03,548 [ERROR] sarracenia.transfer.https __open__ Server couldn't fulfill the request. Error code: 429, Too Many Requests
2024-02-13 22:30:03,548 [WARNING] sarracenia.flow download failed to write shc_20240213_2029_15540.json: HTTP Error 429: Too Many Requests
2024-02-13 22:30:03,548 [INFO] sarracenia.flow do_download attempt 1 failed to download https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/shc_20240213_2029_15540.json to /apps/sarra/public_data/20240213/SHC-REST/20/shc_20240213_2029_15540.json
2024-02-13 22:30:03,548 [WARNING] sarracenia.flow do_download downloading again, attempt 2
2024-02-13 22:30:03,860 [WARNING]

It would be good to put some smarts in the https downloader to back off for a bit when this error is received.

[andreleblanc11]

Have a look at 'e50e8e4b9a70f6a55de5e1898b10e9b6597f3afb' commit in the branch 'improve_airnow_poll'. I think the new HTTP(s) logic in there might be the solution.

You could change the 'max_retry' value. The retries are done via exponential backoff.

[petersilva]

code snippet from @andreleblanc11

                # Setup an HTTP session. Set it up where we can retry 3 times if ever it fails and have exponential backoff applied.
               # Backoff_factor set to 1 so normal exponential backoff.
                session = requests.Session()
                retry = Retry(connect=3, backoff_factor=1)
                adapter = HTTPAdapter(max_retries=retry)
                session.mount('http://', adapter)
                session.mount('https://', adapter)
                resp = session.get(URL)