[BUG] max-wait-time option is being ignored

[pawelmalak]

• I am reporting a bug.
• I am running the latest version of BDfR (v2.7.0 from development branch)
• I have read the Opening an issue

Description

--max-wait-time option is being ignored. I've set it to 10, but it still waits 60 seconds on each error.

Command

bdfr download \
  "data/subreddits/$1" \
  --config config.cfg \
  --subreddit $1 \
  --sort top \
  --time year \
  --verbose \
  --skip html \
  --skip txt \
  --skip-domain youtube.com \
  --skip-domain youtu.be \
  --skip-domain vimeo.com \
  --skip-domain archive.org \
  --file-scheme "{DATE}_{TITLE}_{POSTID}" \
  --folder-scheme . \
  --exclude-id-file "data/ids/ids-$1.txt" \
  --log "data/logs/year_$1.log" \
  --max-wait-time 10

In the command --max-wait-time is set to 10.

Local config.cfg file which is passed to the command also has max_wait_time set to 10:

[DEFAULT]
client_id = <redacted>
client_secret = <redacted>
scopes = identity, history, read, save, mysubreddits
backup_log_count = 3
max_wait_time = 10
time_format = ISO
user_token = <redacted>

Deafult default_config.cfg at C:\Users\Admin\AppData\Local\BDFR\bdfr also has it set to 10:

[DEFAULT]
client_id = <redacted>
client_secret = <redacted>
scopes = identity, history, read, save, mysubreddits
backup_log_count = 3
max_wait_time = 10
time_format = ISO
user_token = <redacted>

Environment (please complete the following information)

• OS: Windows 10
• Python version: 3.10.4
• BDfR: v2.7.0 from development branch

Logs

[2024-03-09 13:41:27,586 - bdfr.downloader - DEBUG] - Attempting to download submission X1
[2024-03-09 13:41:27,588 - bdfr.downloader - DEBUG] - Using Imgur with url http://imgur.com/U1
[2024-03-09 13:41:28,083 - bdfr.resource - WARNING] - Error occured downloading from https://i.imgur.com/U1, waiting 60 seconds: Response code 429
[2024-03-09 13:42:28,088 - bdfr.resource - ERROR] - Max wait time exceeded for resource at url https://i.imgur.com/U1
[2024-03-09 13:42:28,088 - bdfr.downloader - ERROR] - Failed to download resource https://i.imgur.com/U1 in submission X1 with downloader Imgur: Could not download resource: Response code 429

[2024-03-09 13:42:28,089 - bdfr.downloader - DEBUG] - Attempting to download submission X2
[2024-03-09 13:42:28,091 - bdfr.downloader - DEBUG] - Using Imgur with url http://imgur.com/U2
[2024-03-09 13:42:28,813 - bdfr.resource - WARNING] - Error occured downloading from http://imgur.com/U2, waiting 60 seconds: Response code 429
[2024-03-09 13:43:28,825 - bdfr.resource - ERROR] - Max wait time exceeded for resource at url http://imgur.com/U2
[2024-03-09 13:43:28,825 - bdfr.downloader - ERROR] - Failed to download resource http://imgur.com/U2 in submission X2 with downloader Imgur: Could not download resource: Response code 429

[2024-03-09 13:43:28,826 - bdfr.downloader - DEBUG] - Attempting to download submission X3
[2024-03-09 13:43:28,827 - bdfr.downloader - DEBUG] - Using Imgur with url http://imgur.com/U3
[2024-03-09 13:43:29,560 - bdfr.resource - WARNING] - Error occured downloading from http://imgur.com/U3, waiting 60 seconds: Response code 429
[2024-03-09 13:44:29,570 - bdfr.resource - ERROR] - Max wait time exceeded for resource at url http://imgur.com/U3
[2024-03-09 13:44:29,570 - bdfr.downloader - ERROR] - Failed to download resource http://imgur.com/U3 in submission X3 with downloader Imgur: Could not download resource: Response code 429

As you can see, on every error, BDfR waits for 60 seconds: Error occured downloading from URL, waiting 60 seconds

[BluesFanUK]

Same issue, is this fork dead?

[Serene-Arc]

It's not dead, I've just been busy with the semester of late. I will try to get to this soon.