Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add multithreading #463

Open
feefladder opened this issue Mar 5, 2024 · 2 comments
Open

add multithreading #463

feefladder opened this issue Mar 5, 2024 · 2 comments

Comments

@feefladder
Copy link

When downloading all likes or a playlist, I think the program would benefit from a multithreaded approach. Something like adding an option --num-threads n that would partition the list in nths:

main thread is 0

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 ^ 1 ^ 1 ^ 1 ^ 1 ^ 1  ^  1  ^  1  ^  <-- 2 threads
  2   2   2   2   2    2     2     2

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 ^ ^ ^ 1 ^ ^ ^ 1 ^ ^  ^  1  ^  ^  ^  <-- 4 threads
  2 | |   2 | |   2 |  |     2  |  |
    3 |     3 |     3  |        3  |
      4       4        4           4

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 ^ ^ ^ ^ ^ ^ ^ 1 ^ ^  ^  ^  ^  ^  ^  <-- 8 threads
  2 | | | | | |   2 |  |  |  |  |  |
    3 | | | | |     3  |  |  |  |  |
      4 | | | |        4  |  |  |  |
        5 | | |           5  |  |  |
          6 | |              6  |  |
            7 |                 7  |
              8                    8

Of course, this would create a little bit of a data race with the already-downloaded-file, but that could be solved by creating n tempfiles that are later appended to the already-downloaded file (duplicates in playlists are not allowed on soundcloud:
image proof
)

@0pcom
Copy link

0pcom commented Mar 31, 2024

I'm not sure if the files are large enough to really make a difference, but another idea along those same lines is HTTP range requests - if the server supports RFC 7233 (partial content), split the file into chunks, then start downloading and merging the chunks into the destination file concurrently.

an examplre of this is here https://github.com/melbahja/got

@feefladder
Copy link
Author

the thing is about parallelly downloading multiple files from a playlist concurrently, not chunk up files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants