Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it async? [feature request] #249

Open
mpr1255 opened this issue Jun 19, 2023 · 2 comments
Open

Make it async? [feature request] #249

mpr1255 opened this issue Jun 19, 2023 · 2 comments

Comments

@mpr1255
Copy link

mpr1255 commented Jun 19, 2023

I guess development here is no more -- but making this async would be really good/functional. Trying to backup a site with 175k links for example, I'm on URL 80 after a minute. Not really feasible.... I'm going to do something in python but it will be very basic and not a really great CLI like this.

@mpr1255 mpr1255 changed the title Make it async? Make it async? [feature request] Jun 19, 2023
@EvanNotFound
Copy link

you can use -c to download multiple files at a time, it will be way faster.

e.g. -c 20 means 20 files at one time

@sww1235
Copy link

sww1235 commented Nov 20, 2023

A side note: the internet archive has implemented a rate limit feature which probably breaks all mutli-threaded downloads, like -c, since it breaks single threaded downloading as well without substantial delays between requests. See pull request #268 and #266 for more details

https://archive.org/details/toomanyrequests_20191110

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants