libtorrent does not handle case sensitive torrents

[axet]

Please provide the following information

libtorrent version (or branch): 2.0.8

platform/architecture: debian bookworm

compiler and compiler version: https://tracker.debian.org/pkg/libtorrent-rasterbar

qbittorrent failed to save / resume torrent with duplicate filenames (case sensitive) with following error:

13.02.2024 09:50 - Failed to restore torrent. Files were probably moved or storage isn't accessible. Torrent: "TheException". Reason: "TheException fast resume rejected. file_stat(/media/axet/1TB/Games/TheException/TheException/readme.1.txt): mismatching file size"

1. create torrent with "ReadMe.txt" and "readme.txt" in the same folder.
2. restart app.

[arvidn]

since some filesystems are case insensitive, libtorrent detects filenames that would collide on such systems and renames the duplicate files or directories. This is a feature that enables Mac and Windows users download torrents that contain filenames such as your example.

Is there a strong case to be made to support these kinds of torrents on filesystems that have case sensitive names?

[axet]

It looks like a bug, it took some time before I figured out why torrent failed to rehash. Like a while...

If user create (able to create) case sesitive torrents, then he knows what is he doing. Software should properly works on such OS/filesystems. Even windows has support for case sensitive fs right now and this issue could raise on windows as well. I do not see why should we ignore case sensitive FS, because of old windows implementations.

Here a few ways to detect case insensitive FS. I do not this this is a problem.

User can be informed with a simple error, or renaming a file (as it currently implemented) on such FS.

[arvidn]

By default, both windows and MacOS are case insensitive. I think it's fair to say it's a niche feature.

The issues on the opposite side are:

1. increased complexity in detecting whether the filesystem supports case sensitive filenames. The logic to rename duplicate files happens long before any knowledge about where the user intends to save the torrent.
2. Altering behavior based on (subtle) external factors such as whether the filesystem supports case sensitive filenames can be equally surprising to users. At least the current behavior is consistent across platforms and filesystems.
3. moving the storage from one drive to another, or from one system to another will work, because the lowest common denominator is assumed.

So, the issue you highlight could also be addressed with clearer documentation and/or warning messages.

The most reasonable ways, in my mind, would be to:

1. improve documentation, probably of all filename transformations that occur
2. add a flag to disable some (or perhaps all) filename transformations. One has to be careful here though, as there are security risks involved.
3. Issue some kind of feedback when creating a torrent that would cause files to be renamed. This indicates you can't turn around and start seeding such torrent. However, I can't think of a good channel for warning messages like this. Can you?

[axet]

Current implementation is confusing.

Much easier for the end user would be to see a not a warning but an error message when user open a torrent file with case sensitive names and failed to load such torrent.

Easy to implement, clear statement that this project does not support case sensitive torrents. No confusion / expectations.

[arvidn]

Much easier for the end user would be to see a not a warning but an error message when user open a torrent file with case sensitive names and failed to load such torrent.

My impression from users is that they almost always would like to actually download the torrent, no matter how odd it is. Do you think your view of not be allowed or it not being supported is representative of most users?

[axet]

I see your point - it is hard to cover all possible complications caused by support case sensitive names. And I agree, a lot to code for that simple issue. In your example: having cross storage support while moving from case sensitive to insensitive is a lot of work and possible more issues and bugs. Maybe even requires to rework a lot of code and even API of the library.

Don't get me wrong. I still want to see this feature to be implemented. But since current implementation is very confusing and not clear, better to be straight with user, make make matter simple, then confuse them with hidden file renaming. That why I suggest to raise a error instead a warning, as fast and proper solution for that issue.

Your last example - download torrent, does not cover all use cases. And maybe good for most users, also less experienced users, who just only want to download a file, but at the same you are breaking expectation how files should be managed on file sensitive FS. Just because default Mac and Windows installation has case insensitive fs, does not mean it should be default for everyone.

Right now, I suggest to block using such torrent from this library until feature get properly implemented and fixed. Simple and strait is how we get better with code, libraries and utilities.

[arvidn]

do you have a suggestion or vision what a proper solution would look like?

For example, if there was a flag where you could disable all (or most) filename renaming), would that solve the problem in your mind?

When the file-renaming feature is disabled, you defer issues until the first time you try to read or write the file, and you get an I/O error instead. Or, in the case of case-sensitive names referring to the same file, you get data corruption and are banned from the swarm. That's not an excellent outcome.

[axet]

Maybe you can set that mandatory flag per storage when it created, so app would help libtorrent to behave properly with different filesystems.

But that would require API change.

[axet]

Libtorrent can try to detect case sensitive FS / storage. Here is a very clever suggestion by author of https://github.com/privatenumber/is-fs-case-sensitive using invert case.

• https://github.com/privatenumber/is-fs-case-sensitive/blob/develop/src/index.ts

[arvidn]

detecting whether a filesystem supports case sensitive filenames or not is not the main challenge.
The main challenge is to come up with rules and behaviors that are intuitive and unlikely to cause confusion and likely to successfully download torrents.

The file renaming, today, happens when loading/parsing the torrent file. At this point there is no knowledge about which filesystem the user intends to save this torrent to (if at all).

You could argue that perhaps the file renaming should happen at the disk layer, perhaps even being invisible to the user. That comes with its own set of issues. For example:

1. the filenames will show up differently to the user than on disk (because the file_storage will still have the original files)
2. You may end up with security issues, like setting left-to-right text direction and have the filename extension end up in the middle of the filename (it wouldn't on disk, but it would to the user)
3. You may have filenames with invalid paths, such as ".." in the user interface, but not on disk.

Since the file renaming would happen based on some external property (in this case, whether a filesystem is case sensitive) the filename renaming may be less robust, and as long as that logic is improved, file renaming will change and break downloads.

Another property that would be a natural extension would be to only consider filename characters invalid if they truly are invalid on the specific filesystem the files are being saved to, rather than using the lowest common denominator of valid characters. This would (likely) also be slightly less robust as filesystem detection is improved over time.