App irreparably frozen after accidentally queueing up a very large download

[JohnnyBroccoli]

Nicotine+ version: 3.3.2

Operating System/Distribution: macOS Monterey Version 12.7.3

Describe the bug

Nicotine+ app has been rendered completely useless via the spinning beach ball of death no matter how many times I force quit and restart it after accidentally starting up a download of an absolutely massive folder

Expected behavior

I'd expect to be able to clear all downloads and actually use the app like normal

Steps to reproduce the bug

The bug is omnipresent since this morning and doesn't go away regardless of how many times I force quit and restart the app

Additional context

Screenshots, logs, stacktraces or relevant information.

I'm not sure what would be relevant or helpful, so please let me know if I can share anything that might help someone troubleshoot this issue with me (having said that, I can't do much navigating around the app as it is frozen)

[slook]

As a workaround you could edit or delete the downloads.json file to clear the problem downloads, it is usually located inside a hidden folder that is in somewhere like /home/<logon user name>/.local/share/nicotine/.

In the long-term there is discussion to mitigate this kind of problem #1881 (comment) and followup from #1879 (comment) so it's worth keeping a backup of your old downloads.json for future testing. Ideally it would be possible to remedy the issue from within the app.

[mathiascode]

Could you send your downloads.json file to my email address (on my GitHub profile)? It would be interesting to know exactly how many downloads there are. You can then remove the file, and the problem should go away.

To find the folder containing the file:

1. In Finder, Go -> Home
2. Press Command+Shift+. (period) to show hidden folders
3. Go to .local/share/nicotine

Also, how did you end up with so many downloads? There should've been a confirmation dialog before starting a folder download if there are too many files in it.

As far as I understand, downloads still work, and Nicotine+ only freezes when trying to open the Downloads tab?

[slook]

Nicotine+ only freezes when trying to open the Downloads tab?

And of course that includes if the open last remembered tab on startup option is enabled and that tab is the Downloads tab, hence the application window is totally bricked, because not even switching to a different tab is possible under this circumstance.

[JohnnyBroccoli]

Could you send your downloads.json file to my email address (on my GitHub profile)? It would be interesting to know exactly how many downloads there are. You can then remove the file, and the problem should go away.

To find the folder containing the file:

1. In Finder, Go -> Home

2. Press Command+Shift+. (period) to show hidden folders

3. Go to .local/share/nicotine

Also, how did you end up with so many downloads? There should've been a confirmation dialog before starting a folder download if there are too many files in it.

As far as I understand, downloads still work, and Nicotine+ only freezes when trying to open the Downloads tab?

First of all: thank you for your help.

I have found the "download.json" file both you and slook mentioned. Unfortunately, Google wouldn't let me send it because of the size (247.5 MB); if you have any ideas of another way to get you the info, please let me know. As of now, I have made a duplicate of the file and saved it to my desktop and am planning on deleting the original file shortly after finishing this post.

I'm pretty sure it was 100,000+ files. I actually saw the warning before the download started but had a total brain fart and clicked "yes" instead of "no". I'm usually pretty good with this kind of stuff, so I felt kinda dumb about that. Then the app froze up moments later and I felt even dumber.

As to your last paragraph here: downloads do not still work. Nicotine+ might freeze up regardless but will 100% freeze up as soon as I try to click on the "downloads" tab.

[JohnnyBroccoli]

Welp....I deleted the "downloads.json" file from the hidden folder as suggested and the app still freezes up the moment I click on the downloads tab.

Should I delete the "downloads.json.old" file too? Anything else I can try?

[slook]

wouldn't let me send it because of the size (247.5 MB); if you have any ideas of another way to get you the info, please let me know

Maybe put the json file somewhere in your shares, then use the "Copy URL" function in Browse Shares right-click context menu to get a "slsk://..." link that you can put in the email. Alternatively, just put it somewhere obvious and tell Mat what your username is via email.

I delete the "downloads.json.old" file too?

Iirc it falls back to that if the main file is missing, so yes that probably is causing the hang up. You can safely delete all the .json .dbn .old files in that folder and the program will create them again as required. Just don't delete your config file.

had a total brain fart and clicked "yes" instead of "no". ... Then the app froze up moments later and I felt even dumber.

Lol! dumb things happen sometimes, so the program shouldn't give the possibility to end up in such a dumb situation.

[JohnnyBroccoli]

Thank goodness.

Deleting the "downloads.json.old" folder as well has fixed the freezing issue. If you guys would still like me to drop the "downloads.json" and/or "downloads.json.old" files in to my shared folder and send you my Nicotine+ user name via email, please let me know (especially if it'll help y'all improve this great app and/or help others avoid this issue in the future).

I really appreciate the time both of you put in to helping me with this. Very unlikely I would have figured any of this out on my own.

[JohnnyBroccoli]

On a related tip, I believe there's an option in settings to limit the amount of downloads other users can snag from you (aka uploads) at once. Maybe adding a similar option for downloads would be helpful.

Also....to the user I accidentally initiated these downloads from: sorry! I hope my blunder hasn't screwed up your ability to use the app and if it has, I hope you've found your way here for the solution.

[mathiascode]

I have found the "download.json" file both you and slook mentioned. Unfortunately, Google wouldn't let me send it because of the size (247.5 MB); if you have any ideas of another way to get you the info, please let me know. As of now, I have made a duplicate of the file and saved it to my desktop and am planning on deleting the original file shortly after finishing this post.

That explains things. There were large performance improvements in 3.3.0, but the number of files is probably in the millions if the downloads.json file is that large.

We should do something in cases where the number of downloads is this ridiculous, but I'm not sure what. Discard downloads? What should the limit be? How is the user informed?

Feel free to send me an email containing your username, and I'll download your downloads.json file.

[JohnnyBroccoli]

That explains things. There were large performance improvements in 3.3.0, but the number of files is probably in the millions if the downloads.json file is that large.

Not quite millions. I believe it was 130,000 or so.

We should do something in cases where the number of downloads is this ridiculous, but I'm not sure what. Discard downloads? What should the limit be? How is the user informed?

Agreed. I'd say something like a 500 file queue limit based on my usage patterns (even before this, I had set a limit on uploads of 200 files at a time for each user within the settings menu). Don't know much about the backend of applications like this but I'd think an automatic private message or notification alerting both the downloader and the uploader that they are "only allowed to have a queue of 500 files or smaller at any given time (as a safety measure to protect all users)" or something.

[slook]

It's plausible that there are use cases for initiating massive transfers, such as buddies who wish to create a mirror, for example.

[mathiascode]

That explains things. There were large performance improvements in 3.3.0, but the number of files is probably in the millions if the downloads.json file is that large.

Not quite millions. I believe it was 130,000 or so.

Looks like it's 1.3 million downloads. Impressive!

[JohnnyBroccoli]

LOL guess I also misread the warning prompt before accidentally clicking on "yes" instead of "no"

[slook]

Perhaps it would help if the dialog button labels were changed to something like "Cancel" and "Download".

[mathiascode]

Perhaps it would help if the dialog button labels were changed to something like "Cancel" and "Download".

Seems like a good idea.

Looks like another case where almost all time is spent adding rows to the treeview, because Gtk.TreeView insertions are quadratic (O(n^2)).

[JohnnyBroccoli]

Perhaps it would help if the dialog button labels were changed to something like "Cancel" and "Download".

Sounds like a good idea.

Maybe also add a little note saying something like "proceed with caution when attempting to download more than x amount of files at a time, as downloads of this size have been known to brick Nicotine+".

[slook]

We should do something in cases where the number of downloads is this ridiculous, but I'm not sure what.

What about implementing Threading and/or Multiprocessing during any potentially long (of more than, say, 1000 files) user-requested operations, such as while loading transfers history file and while batch-adding transfers from a multi-selection. The Tree View container can be hidden while it is busy and replaced with a label or progress bar widget.

What should the limit be?

Ideally there should not be any kind of artificial limit imposed by the user interface, because in a perfect world that should be only limited by the capabilities of the system and network environment. If peers want to exchange millions of files between themselves, then so be it (provided that is truly want they actually intended to do, and both clients are performant enough).

Edit: On second thought, perhaps a sanity check limit would be sensible, in respect of other/older clients that may be limited by technicality up to a defined integer for whatever reason, such as 32768, just to pick a number out of the air.

How is the user informed?

Preferably, display a Progress Bar popup or widget in the Status Bar (similar to the "Scanning Shares" spinner) along with a Cancel [X] button to raise some kind of flag to let the core know that the user wants to abort the requested operation. It would be a bonus if the status spinner label could indicate "Added X of X items" or such like, unless it is indeterminate then "Added X items".

If the status bar is frozen and cannot be clicked on, then a progress window could popup after a certain number of seconds have elapsed (say, 10 seconds) since the beginning of the operation. A button to cancel the current operation could be provided, or otherwise the window need only contain a plain label. A progress bar widget and other decorations could be added later.

Discard downloads?

Such kind of "is_user_wanting_to_abort" flag can then be caught inside the calling core loop to break it. After the loop is broken in this way then any unprocessed downloads would, in effect, be discarded.

[DeathStalker77]

Yeah, I would say any single download request that exceeds 500k files should be prevented, IF the program cannot handle it properly. Definitely over 1 million! Even MY queue has never been that large! :-D

I frequently go well over 150,000 files queued, so I do not it any way believe the limit should be that low. WARNINGS on over 1,000 is fine - maybe adjust the wording/color/format of WARNINGS for larger single requests.

--- DS