Opds connection under docker deployment cannot display books in TXT files

[shaoyangx]

docker deployment:

services:
cops:
image: linuxserver/cops:latest
container_name: cops
network_mode: bridge
environment:
- PUID=1000
- PGID=100
- TZ=Asia/Shanghai
volumes:
- /path/cops:/config
- /path/Book:/books #:ro
ports:
- 8089:80
restart: unless-stopped

config_local.php

$config['calibre_directory'] = '/books/ebooks/';

$config['calibre_internal_directory'] = '/books/ebooks/';
$config['cops_title_default'] = "COPS";

$config['cops_use_url_rewriting'] = "0";
$config['cops_x_accel_redirect'] = "X-Accel-Redirect";

$config['cops_thumbnail_handling'] = "";
$config['cops_thumbnail_cache_directory'] = "/config/cache/";

$config['cops_author_split_first_letter'] = '0';
$config['cops_titles_split_first_letter'] = '0';
$config['cops_titles_split_publication_year'] = '0';

$config['cops_prefered_format'] = ['EPUB', 'TXT', 'PDF', 'AZW3', 'AZW', 'MOBI', 'CBR', 'CBZ'];

Other configurations have not changed

All books can be displayed normally in the WEB. When connected to opds, the tags and number of authors can be displayed（(Contains author, tag in TXT format)）. Books in TXT format cannot be opened, only epub format can be opened.

Android APP：Moon+ Reader 9.3 ,FBReader
Books in TXT format all show that the directory is empty

[mikespub]

Hi @shaoyangx sorry to hear that. This is probably due to a missing mimetype in lib/Calibre/Data.php

You can check this in your browser via /feed.php?page=10 or similar - you probably don't get an "acquisition" link for TXT files, but you do get an "acquisition" link for EPUB files:

<link href="/cops/fetch.php?id=17&amp;type=epub&amp;data=20" type="application/epub+zip" rel="http://opds-spec.org/acquisition" title="EPUB"/>

With the change I committed above, you should get an "acquisition" link for TXT files as well:

<link href="/cops/fetch.php?id=17&amp;type=txt&amp;data=22" type="text/plain" rel="http://opds-spec.org/acquisition" title="TXT"/>

[shaoyangx]

Thank you very much, after modifying lib/Caliber/Data.php

FBReader can already display txt format books normally.

However, Moon+ Reader 9.3 can display the book list, but cannot open it. clicking on a book, only the cover image of the book will be displayed.I can’t see the book information, author, introduction, etc., and I can’t download it either.
Even if I use$config['cops_generate_invalid_opds_stream'] = '1';
（Moon+ Reader can display content normally using caliber content server，But caliber takes up too much memory）

[mikespub]

I know there were some issues with Moon+ Reader if you didn't specify $config['cops_full_url'] in your config_local.php - see issue #66

For example:

$config['cops_full_url'] = 'http://localhost/cops/';

And apparently Moon+ Reader also planned to restrict OPDS feeds to be https only in newer versions, so that might be part of the problem too, but it's hard to say without being able to reproduce.

Could you show the XML output of one of the "book list" feeds that give you problems, and one that doesn't give you problems?
There's no such thing as a "book details" page in the OPDS feeds, so Moon+ Reader must be having issues with the book list to show the cover instead of the details...

[shaoyangx]

feed.zip
This is feed.php
This includes an EPUB format book that can be accessed normally, and an accessible TXT format book that can only display pictures.

【Could you show the XML output of one of the "book list" feeds that give you problems, and one that doesn't give you problems?】
I don’t quite understand this sentence. I tested it on two different apps. Currently, all books in TXT format cannot be viewed in Moon+ Reader 9.3(epub is normal), and FBreader can be used normally.
If you need, I can set up a library connection with the same parameters on the VPS for testing.

$config['cops_full_url'] =
I cannot use this configuration normally, and the following three URLs cannot be used normally.
http://192.168.31.114/cops/ Deploy Machine IP
http://192.168.31.114:8089/cops/ Docker deployment mapped port
http://localhost/cops/

[mikespub]

Thanks - the XML output in the .zip file was exactly what I was looking for. Since I don't all the apps available for testing, I wanted to check if what COPS generated as OPDS feed for you was at least correct.

As I can see, there is no difference between the OPDS entries for EPUB files and TXT files other than the expected type="application/epub+zip" instead of type="text/plain" in the "acquisition" links, and it works for FBReader.

So I suspect that Moon+ Reader expects some special mimetype or format for TXT entries - I'm afraid you'll have to ask them to see if they can clarify what would need to be changed...

Perhaps @SenorSmartyPants can point you to the right place to ask since he's been in contact with them too?

[shaoyangx]

Sorry, I used Google Translate to communicate. I may have misunderstood some of the meaning.
I don't know much about the technical aspects, but it suddenly occurred to me that it was normal for me to use caliber's opds service before. Maybe can see the problem from caliber's documentation？

This is the opds generated by Caliber, which can be used normally in both Moon+ Reader and FBreader.
calibre.zip

[mikespub]

No problem, and that was a good idea. Unfortunately based on the difference in content it's not immediately clear to me why Moon+ Reader should decide to show EPUB correctly and TXT not.

At least we can exclude the mimetype - they're the same for both. But is it because there's a length="..." specified, or because there's no image, or...?

Without someone familiar with the Moon+ Reader code it's hard to guess what might be the reason - and I don't think it would make sense to try out different things blindly, sorry :-(

[shaoyangx]

Thank you very much. I like cops very much. I have always wanted to find a lightweight opds container. I used caliber-web, which took up 150Mib, but cops only uses less than 20Mib to achieve almost the same function, which is completely satisfactory. meet my functional requirements.
It seems I should try to contact the developers of Moon+ Reader.

非常感谢，我非常喜欢cops，曾经我一直想要找到一个轻量化的opds容器，以前我使用calibre-web，占用足有150Mib，而cops仅使用不到20Mib就达到了近乎相同的功能，完全满足了我的功能需求。
看来我应该尝试联系下Moon+ Reader的开发者。

[mikespub]

Glad you like it - that's why @seblucas made it and we continue to maintain it :-)

I prepared a new COPS release 2.5.6 to trigger a new release for the linuxserver/cops docker image.

It includes the change to lib/Calibre/Data.php for TXT files, and some other OPDS changes that might help (or not).

Let us know if you hear any more details from Moon+ Reader team

[shaoyangx]

Unfortunately, I did not get a reply from Moon+ Reader.

I also tried to search for information about opds (https://specs.opds.io/opds-1.2), but I didn’t find what I wanted to find, although I didn’t understand it very well in the end...

I used some simple tests of my own to find out the differences.
I used Visual Studio Code, edited a local xml file, and generated the web through the yuichinukiyama.vscode-preview-server plug-in for testing. (I guess there should be no problem with this method)

The test results feel weird，But from a layman's perspective, I see that the problem lies in this one statement.

Original: Unrecognized.
<link href="/fetch.php?id=2&amp;type=txt&amp;data=2" type="text/plain" rel="http://opds-spec.org/acquisition" title="TXT" length="5442482" mtime="2024-05-05T11:06:33+00:00"/>

However, if you add /get/txt before the connection, no other parameters need to be changed, and it will be recognized normally.
<link href="/get/txt/fetch.php?id=2&amp;type=txt&amp;data=2" type="text/plain" rel="http://opds-spec.org/acquisition" title="TXT" length="5442482" mtime="2024-05-05T11:06:33+00:00"/>

Or, add .txt directly after the URL, and it can also be recognized and displayed.
<link href="/fetch.php?id=2&amp;type=txt&amp;data=2.txt" type="text/plain" rel="http://opds-spec.org/acquisition" title="TXT" length="5442482" mtime="2024-05-05T11:06:33+00:00"/>

Or modify the type directly, and it will directly become identifiable...
<link href="/fetch.php?id=2&amp;type=txt&amp;data=2" type="application/epub+zip" rel="http://opds-spec.org/acquisition" title="TXT" length="5442482" mtime="2024-05-05T11:06:33+00:00"/>

I can only reluctantly come to the conclusion that perhaps Moon+ Reader uses a method when detecting type="application/epub+zip", and detects type="text/plain" and uses another parsing method?

This problem is no longer caused by copd, but by Moon+ Reader, If can't get an answer from the Moon+ Reader developer, I'm afraid it won't be easy to solve this problem.

[mikespub]

Indeed this isn't the first time Moon+ Reader had some unexpected behavior compared to other readers. Thanks for testing it out, and hopefully you'll hear something from the Moon+ Reader team...

In the meantime you could try enabling the url rewriting feature in config_local.php by changing:

$config['cops_use_url_rewriting'] = "1";

You will need to adapt the /path/cops/nginx/site-confs/default.conf file as well:

    #Useful only for Kobo reader
    #location /cops/ { - fix wrong location for /download/ in default.conf
    location /download/ {
        rewrite ^/download/(\d+)/(\d+)/.*\.(.*)$ /fetch.php?data=$1&db=$2&type=$3 last;
        rewrite ^/download/(\d+)/.*\.(.*)$ /fetch.php?data=$1&type=$2 last;
        break;
    }

This should turn all book links to something like /download/123/This%20is%20my%20file.txt which Moon+ Reader should be able to recognize.

Or you could use the last test you did by changing the lib/Calibre/Data.php file in COPS again, and replace "text/plain" with "application/epub+zip" for txt files in the list of mimetypes, as a way to "force" Moon+ Reader.

[mikespub]

Pull request linuxserver/docker-cops#52 to update default nginx site conf

[mikespub]

Hi @shaoyangx

with the latest linuxserver/cops docker image you should be able to download TXT and EPUB files with Moon+ Reader.

Changes required in config files:

1. in /path/cops/config_local.php

$config['cops_use_url_rewriting'] = "1";

2. in /path/cops/nginx/site-confs/default.conf

See changes in linuxserver/docker-cops#52

If that still doesn't work, please post the generated OPDS output and what happens with Moon+ Reader here, and we'll try again once you hear back from them...

[dunxd]

@mikespub this is leading me to think that I need to make URL Rewriting either an option in HA COPS or just turned on, as I can't say what readers users might be using.

Is there any reason why having URL Rewriting set to on should not be the default? Does this break things for more common readers?

[dunxd]

Though since HA-COPS uses PHPs built in web server I would have to do it something like this.

[shaoyangx]

Thank you very much. In the new version, after turning on $config['cops_use_url_rewriting'] = "1";, The problem was perfectly solved,the txt format can be accessed normally.
last night, I also received a reply from the Moon+ Reader developer and stated that this problem will be fixed in the next version.
The following is the original reply:

你好, 找到原因了, 对txt格式(type为text/plain时)的解析加了不必要的约束条件(url里要有/txt部分)
可能是以前的OPDS的txt格式下载链接都归类到这里面, 很久以前不知道是什么原因, 多加了这一个不必要的判断, 下个版本去掉这里就可以了
txt格式支持这三种: type为txt或text/plain, url链接以.txt结束 (你很厉害, 三种情况都找到了👍👍👍)
Thank you!
Regards
Seany

Googel translate：
Hello, I found the reason. Unnecessary constraints are added to the parsing of txt format (when type is text/plain) (there must be a /txt part in the url)
It may be that the previous txt format download links of OPDS were classified here. I don’t know why a long time ago, this unnecessary judgment was added. I will just remove it in the next version.
The txt format supports these three types: type is txt or text/plain, and the url link ends with .txt (you are awesome, you found all three situations👍👍👍)
Thank you!
Regards
Seany

[shaoyangx]

@mikespub this is leading me to think that I need to make URL Rewriting either an option in HA COPS or just turned on, as I can't say what readers users might be using.

Is there any reason why having URL Rewriting set to on should not be the default? Does this break things for more common readers?

I personally think that it does not need to be turned on by default, even if it is not turned on: $config['cops_use_url_rewriting'] = "1";
The resulting connection is also a standards-aware connection.
At present, all mainstream clients that support opds can parse normally. Even Moon+ Reader is only caused by non-standard parsing, and the developer also stated that this problem will be fixed in the next version.
$config['cops_use_url_rewriting'] = "1"; I think it can be used as an alternative.To prevent new clients from having the same problem in the future

[mikespub]

That's excellent news @shaoyangx thanks. Glad it's working out for you now - good detective work :-)

And @dunxd I also think that the current use_url_rewriting should be kept as an option, not as the default. It requires a custom configuration for each type of webserver out there, and it is also not aligned with the more generic use_route_urls that would apply to all generated URLs, not only download/view links in OPDS.

I haven't activated that option by default either because it doesn't cover all COPS endpoints yet and it also may require webserver configuration, but at least it's based on standard PATH_INFO processing like most PHP front-end controllers out there, and not custom rewrite rules for each type of link. Some of the newer endpoints already use that by default, like REST API and OPDS 2.0 (dev only).

Note: having said that, if in HA COPS you would prefer to enable use_url_rewriting by default for now, I don't have any issues since there you can control both the initial COPS config and the webserver config.

[dunxd]

I understood the Wiki page on URL rewriting as saying that this feature existed to support being able to download files to the Kobo eReader. Have I understood correctly? I don't have a Kobo to verify this myself.

I would like HA COPS to support as many devices as possible. However, I don't want to figure out URL rewriting unless this is verified as required for popular devices, especially since it probably requires including a different web server in HA COPS which I don't want to do just now.

[mikespub]

Hi @dunxd

from the project history at seblucas/cops, URL rewriting was on by default in the 0.1.x version but then switched off by default starting in 0.2.x version and later, so about 12 years ago now :-)

The only devices who seem to have trouble downloading books via regular fetch.php?data=...&type=... links are OPDS readers with "strange" parsing rules - mainly Kobo and Moon+ Reader that I know of, but there may be others.

You could probably use the router script trick with the built-in web server you referred to earlier to handle those rewrites if need be. There's a new router.php file I added recently that could be re-purposed for this if you're interested - right now it doesn't do much besides pointing to a (future) front-end controller that doesn't work yet...

[mikespub]

Closing issue as resolved using url rewriting for now. If/when Moon+ Reader can handle TXT files without it we can revisit