Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Very slow to combine chunks #4421

Open
DrekoDev opened this issue Jan 6, 2024 · 4 comments
Open

Very slow to combine chunks #4421

DrekoDev opened this issue Jan 6, 2024 · 4 comments
Assignees
@GammaC0de
Labels
bug Something isn't working pyLoad Stable

Comments

@DrekoDev
Copy link

DrekoDev commented Jan 6, 2024 •
edited

Description

Hi

I use PyLoad via a seedbox service (seedbox.fr).

It works very well for downloading videos, but it takes a lot of time to combine different chunks. I recently downloaded a movie and it took more than 20 minutes to combine to finalize the download.

I had to set the "Maximum connections for one download" setting to 1 to decrease that length of time, because before that, it took half an hour.

Debug log

06.01.2024 16:12:34 | INFO | ADDON UpdateManager: All plugins are up to date!
06.01.2024 16:12:40 | INFO | ADDON ClickNLoad: Proxy listening on 0.0.0.0:9666
06.01.2024 19:45:33 | INFO | Added package Equalizer containing 1 links
06.01.2024 19:45:33 | DEBUG | Redirected import module.plugins.internal.SimpleHoster -> userplugins.internal.SimpleHoster
06.01.2024 19:45:34 | INFO | Download starts: https://1fichier.com/?4en40rf9uxeoc9ik02sf&af=4814702
06.01.2024 19:45:34 | DEBUG | ADDON ExternalScripts: No script found under folder download_preparing
06.01.2024 19:45:34 | DEBUG | ADDON UserAgentSwitcher: Setting connection timeout to 60 seconds
06.01.2024 19:45:34 | DEBUG | ADDON UserAgentSwitcher: Setting maximum redirections to 10
06.01.2024 19:45:34 | DEBUG | ADDON UserAgentSwitcher: Use custom user-agent string Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0
06.01.2024 19:45:34 | DEBUG | HOSTER OneFichierCom[58]: Plugin version: 1.19
06.01.2024 19:45:34 | DEBUG | HOSTER OneFichierCom[58]: Plugin status: testing
06.01.2024 19:45:34 | WARNING | HOSTER OneFichierCom[58]: Plugin may be unstable
06.01.2024 19:45:34 | DEBUG | ACCOUNT OneFichierCom: Reached login timeout for user dre*******
06.01.2024 19:45:34 | INFO | ACCOUNT OneFichierCom: Relogin user dre*******...
06.01.2024 19:45:34 | DEBUG | ACCOUNT OneFichierCom: LOAD URL https://1fichier.com/login.pl?lg=en | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=False
06.01.2024 19:45:34 | WARNING | ACCOUNT OneFichierCom: Skipped login user dre******* | Already signed in
06.01.2024 19:45:34 | INFO | ACCOUNT OneFichierCom: Grabbing account info for user dre*******...
06.01.2024 19:45:34 | DEBUG | ACCOUNT OneFichierCom: LOAD URL https://1fichier.com/console/abo.pl | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=False
06.01.2024 19:45:34 | DEBUG | ACCOUNT OneFichierCom: Expire date: 2024-07-23
06.01.2024 19:45:34 | DEBUG | ACCOUNT OneFichierCom: Account info for user dre*******: {'login': {'timestamp': 1704570334.050134, 'password': '', 'stats': [2, 0], 'valid': True}, 'data': {'validuntil': 1721689200.0, 'plugin': , 'trafficleft': -1, 'maxtraffic': None, 'premium': True, 'login': 'dre', 'type': 'OneFichierCom', 'options': {'limitDL': ['0']}}}
06.01.2024 19:45:34 | DEBUG | ACCOUNT OneFichierCom: Using account dre****
06.01.2024 19:45:34 | INFO | HOSTER OneFichierCom[58]: Processing url: https://1fichier.com/?4en40rf9uxeoc9ik02sf&af=4814702
06.01.2024 19:45:34 | INFO | HOSTER OneFichierCom[58]: Looking for direct download link...
06.01.2024 19:45:34 | DEBUG | HOSTER OneFichierCom[58]: LOAD URL https://1fichier.com/?4en40rf9uxeoc9ik02sf | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=True
06.01.2024 19:45:34 | DEBUG | HOSTER OneFichierCom[58]: Redirect #1 to: https://c-1.1fichier.com/p916877324
06.01.2024 19:45:34 | DEBUG | HOSTER OneFichierCom[58]: LOAD URL https://c-1.1fichier.com/p916877324 | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=True
06.01.2024 19:45:35 | INFO | HOSTER OneFichierCom[58]: Direct download link detected
06.01.2024 19:45:35 | INFO | HOSTER OneFichierCom[58]: Downloading file...
06.01.2024 19:45:35 | DEBUG | HOSTER OneFichierCom[58]: DOWNLOAD URL https://1fichier.com/?4en40rf9uxeoc9ik02sf | cookies=True | resume=None | get={} | disposition=True | chunks=None | post={} | ref=True
06.01.2024 19:45:36 | DEBUG | Content-Disposition: (Name of the movie)
06.01.2024 19:45:36 | DEBUG | Chunked with range 0-
@DrekoDev DrekoDev added bug Something isn't working pyLoad Stable labels Jan 6, 2024
@DrekoDev
Copy link
Author

Hi @GammaC0de , do you have any clue ? If you need more logs here they are

`

7 23.01.2024 18:05:25 DEBUG ADDON UserAgentSwitcher: Use custom user-agent string Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0
38 23.01.2024 18:05:25 DEBUG HOSTER OneFichierCom[240]: Plugin version: 1.19
39 23.01.2024 18:05:25 DEBUG HOSTER OneFichierCom[240]: Plugin status: testing
40 23.01.2024 18:05:25 WARNING HOSTER OneFichierCom[240]: Plugin may be unstable
41 23.01.2024 18:05:25 INFO ACCOUNT OneFichierCom: Grabbing account info for user dre*******...
42 23.01.2024 18:05:25 DEBUG ACCOUNT OneFichierCom: LOAD URL https://1fichier.com/console/abo.pl | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=False
43 23.01.2024 18:05:25 DEBUG ACCOUNT OneFichierCom: Expire date: 2024-07-23
44 23.01.2024 18:05:25 DEBUG ACCOUNT OneFichierCom: Account info for user dre*******: {'login': {'timestamp': 1706032894.382562, 'password': '***', 'stats': [2, 1706033009.192815], 'valid': True}, 'data': {'validuntil': 1721689200.0, 'plugin': , 'trafficleft': -1, 'maxtraffic': None, 'premium': True, 'login': 'dre', 'type': 'OneFichierCom', 'options': {'limitDL': ['0']}}}
45 23.01.2024 18:05:25 DEBUG ACCOUNT OneFichierCom: Using account dre*******
46 23.01.2024 18:05:25 INFO HOSTER OneFichierCom[240]: Processing url: https://1fichier.com/?x1pkrjifk6w5gezkcop0&af=4814702
47 23.01.2024 18:05:25 INFO HOSTER OneFichierCom[240]: Looking for direct download link...
48 23.01.2024 18:05:25 DEBUG HOSTER OneFichierCom[240]: LOAD URL https://1fichier.com/?x1pkrjifk6w5gezkcop0 | redirect=False | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=True
49 23.01.2024 18:05:27 DEBUG HOSTER OneFichierCom[240]: Redirect #1 to: https://c-4.1fichier.com/p947107115
50 23.01.2024 18:05:27 DEBUG HOSTER OneFichierCom[240]: LOAD URL https://c-4.1fichier.com/p947107115 | redirect=False | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=True
51 23.01.2024 18:05:28 INFO HOSTER OneFichierCom[240]: Direct download link detected
52 23.01.2024 18:05:28 INFO HOSTER OneFichierCom[240]: Downloading file...
53 23.01.2024 18:05:28 DEBUG HOSTER OneFichierCom[240]: DOWNLOAD URL https://1fichier.com/?x1pkrjifk6w5gezkcop0 | cookies=True | resume=None | get={} | disposition=True | chunks=None | post={} | ref=True
54 23.01.2024 18:05:29 DEBUG Content-Disposition: Percy.Jackson.and.the.Olympians.S01E06.FRENCH.1080p.WEB.H264-FW-Wawacity.boats.mkv
55 23.01.2024 18:05:29 DEBUG Chunked with range 0-
56 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: File saved
57 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: Checking download...
58 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: File is OK
59 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: Checking file (with built-in rules)...
60 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: Checking file (with custom rules)...
61 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: Checking for link errors...
62 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: No errors found
63 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: No errors found
64 23.01.2024 18:12:05 DEBUG ADDON ExternalScripts: No script found under folder download_processed
65 23.01.2024 18:12:05 DEBUG ADDON ExternalScripts: No script found under folder package_processed
66 23.01.2024 18:12:05 INFO Download finished: Percy.Jackson.and.the.Olympians.S01E06.FRENCH.1080p.WEB.H264-FW-Wawacity.boats.mkv
67 23.01.2024 18:12:05 DEBUG ADDON ExternalScripts: No script found under folder download_finished
68 23.01.2024 18:12:05 INFO Package finished: percy
69 23.01.2024 18:12:05 DEBUG ADDON ExternalScripts: No script found under folder package_finished

`

@GammaC0de
Copy link
Member

GammaC0de commented Jan 23, 2024 •
edited

but it takes a lot of time to combine different chunks

The process of chunks combination is just a matter of simple file copy:

pyload/src/pyload/core/network/http/http_download.py

Lines 88 to 104 in ed850e8

def _copy_chunks(self):
init = self.info.get_chunk_filename(0) #: initial chunk name
if self.info.get_count() > 1:
with open(init, mode="rb+") as fo: #: first chunk file
for i in range(1, self.info.get_count()):
#: input file
#: seek to beginning of chunk, to get rid of overlapping chunks
fo.seek(self.info.get_chunk_range(i - 1)[1] + 1)
fname = f"{self.filename}.chunk{i}"
with open(fname, mode="rb") as fi:
buffer_size = 32 << 10
while True: #: copy in chunks, consumes less memory
data = fi.read(buffer_size)
if not data:
break
fo.write(data)

What is the output of http://<pyload_ip>/info?

Without having access to the seedbox I cannot do anything..

@DrekoDev
Copy link
Author

but it takes a lot of time to combine different chunks

The process of chunks combination is just a matter of simple file copy:

pyload/src/pyload/core/network/http/http_download.py

Lines 88 to 104 in ed850e8

def _copy_chunks(self):
init = self.info.get_chunk_filename(0) #: initial chunk name
if self.info.get_count() > 1:
with open(init, mode="rb+") as fo: #: first chunk file
for i in range(1, self.info.get_count()):
#: input file
#: seek to beginning of chunk, to get rid of overlapping chunks
fo.seek(self.info.get_chunk_range(i - 1)[1] + 1)
fname = f"{self.filename}.chunk{i}"
with open(fname, mode="rb") as fi:
buffer_size = 32 << 10
while True: #: copy in chunks, consumes less memory
data = fi.read(buffer_size)
if not data:
break
fo.write(data)

What is the output of http://<pyload_ip>/info?

Without having access to the seedbox I cannot do anything..

Here is the information tab:

Python:2.7.18 (default, May 3 2020, 20:31:45) [GCC 9.2.0]OS:posix linux2 Linux 7facc51a3a80 3.10.0-1160.102.1.el7.x86_64 #1 SMP Tue Oct 17 15:42:21 UTC 2023 x86_64Version de pyLoad:0.4.20Dossier d'Installation:/app/pyloadDossier de Configuration:/configDossier de Téléchargement:/downloadsEspace libre :152.20 GiBLangue:enWebinterface Theme:modern by [GammaC0de](https://github.com/GammaC0de) based on work of [Marco Fernandes](https://github.com/xunil75) and Dogan BagciPort de l'Interface Web8000Port de l'Interface à Distance7227

Would you need an access to the pyload on my seedbox ?

@GammaC0de
Copy link
Member

Would you need an access to the pyload on my seedbox

To get access just to the pyload on the seedbox would not help because this is a system and OS issue.

You can read here https://unix.stackexchange.com/questions/79751/is-there-a-faster-alternative-to-cp-for-copying-large-files-20-gb for possible reasons why this is happening.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@GammaC0de GammaC0de

Labels
bug Something isn't working pyLoad Stable
Milestone
No milestone
Development

No branches or pull requests
2 participants
@GammaC0de @DrekoDev