Kobo Sync issue

[Dinth]

Describe the bug/problem
Hey. I have set up Kobo Sync as per documentation. Whenever im trying to access the shop on the Kobo im getting "Network Error" message. When i copy and paste the link:
https://calibre.XXXXX.XX/kobo/XXXXXXXXXXXXXXXXXXeb06bbee7867bd1b to a normal internet browser, im only getting:
{}
content. Exactly same page appears when i replace my token with random letters. It's probably me doing something wrong (as calibre-web.log doesnt show any errors), but what i have done wrong?

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Docker container [ None/Technosoft2000/Linuxuser]: linuxuser
  

Additional context
Add any other context about the problem here. [e.g. access via reverse proxy]

[jef]

I also get an empty body, but syncing works fine. I just can't download anything.

[jango]

I think I found an explanation and a workaround, at least in my case. In short, for now, the workaround is to run calibre-web on port 80. If you run docker, you have a better option: change server port to 80 in server configuration and run container as PUID=0 and PGID=0. You can then then map internal port 80 to whatever port you want, for example:

version: "3.7"
services:
  calibre-web:
    image: linuxserver/calibre-web
    container_name: hs_calibre_web
    environment:
      - PUID=0
      - PGID=0
      - TZ=America/Toronto
      - DOCKER_MODS=linuxserver/calibre-web:calibre #*optional* & **x86-64 only**
    volumes:
      - ./data:/config
      - /storage/CalibreLibrary:/books
    ports:
      - 8083:80
    labels:
      - "traefik.http.routers.calibrelan.rule=Host(`calibre.lan`)"
    restart: unless-stopped
    networks:
      - home-services-network

networks:
  home-services-network:
    external: true
    name: home-services-network  

Why this is happening? I used WireShark to look at the transactions and one of the JSON returned by calibre-web for the following request (a book):

http://calibre.lan/kobo/69e9a753a4f12e53d2f65a28122f53eb/v1/library/85e0af05-bd73-4dbd-b46b-d6d67f201cd1/metadata

The response contains a download URL with the port number 8083 which is the default port that Calibre is running on. Note that even though internally in my docker container the app is running on port 8083, from outside, the port is 80. So Kobo was getting a URL that was not accessible:

...
"DownloadUrls":[
         {
            "Format":"EPUB3",
            "Platform":"Generic",
            "Size":2685019,
            "Url":"http://calibre.lan:80/download/622/epub"
         },
         {
            "Format":"EPUB",
            "Platform":"Generic",
            "Size":2685019,
            "Url":"http://calibre.lan:80/download/622/epub"
         }
      ]
...

Changing port to 80 still exposes the port in JSON but at least now the port is accessible. The problematic code is here:

calibre-web/cps/kobo.py

Line 277 in 8608ff1

url_port=config.config_port,

it's either this line should not use the port number at all, or have it configurable, at least for the docker scenario.

@OzzieIsaacs if you give me a direction on how you want this fixed going forward, I can try my hand at submitting a patch. Looks like #1227 is related.

[jef]

Hey @jango. I made a PR earlier today that I believe would address this issue. Wasn't sure which all issues would have this close, so I didn't tag any of them.

#1516

[OzzieIsaacs]

@jef I'm not happy with the additional setting:
If I get it right our problem is that calibre-web is generating wrong urls in the kobo section for downloads and other stuff, but any other links (like on page page, admin section and so on) are generated right.
The kobo reader has the "feature" to generate "abnormal" urls, if it sends a request on a non standard port like 8083, it will not include the port in the request string, so the kobo request would be: http://127.0.0.1/something instead of the correct http://127.0.0.1:8083/something. To work around this issue (which is a problem in non proxied applications, as all proxies I know would make a correct request out of it), the proxied case is detected in reverseproxy.py. The docker is now "mapping" the internal port 8083 to an external port (let's say 80), I guess by some sort of reverse proxying. So Calibre-web would be accessed from outside by http://127.0.0.1 and this is forwardes to http://127.0.0.1:8083.
Now the problem in reverseproxy.py is that only proxies with path in it are detected, and not a port/host forward.
Wouldn't it just work out to add self.proxied=true to line 80 in reverseproxy.py to archive the same result?

[Dinth]

Thanks for looking into this @OzzieIsaacs. I just wanted to say that my setup is proxied and using default 443 port (as it's https there's no need to specify a port).

[jef]

Thanks @OzzieIsaacs! Didn't realize that self accessed this object! A lot easier approach.

[OzzieIsaacs]

It‘s working for you?

[jef]

Haven't pulled it down yet. I'm on vacation. I can try when back home if no one hasn't yet.

[jef]

I'm using the linuxserver docker image, so I'll have to update the image to grab master instead of releases. Once I update that, I can test!

[jef]

Pulled latest down, and remade a URL. Still pointed the urls to the internal port and not the external.

This is also related for those using the Docker image from LinuxServer linuxserver/docker-calibre-web#90 since we are not pulling from a release, but rather master to troubleshoot the issue.

I've got to pull down and build locally to actually test. I'll confirm this when I get back from vacation. I'll keep this issue updated.

[jef]

Hey @OzzieIsaacs, that didn't seem to work. I deleted my old sync token and got a new one and put it on my Kobo Forma. I went to go look at the settings after syncing and it still set it to the 8083 port. It did pull my books and did a proper syncing, but couldn't get pass the download.

[jef]

I used my branch as well for testing. It set up the URLs correctly, but it still didn't download. Not sure what the issue is.

[Dinth]

Im not sure if that's relevant to the issue, but whenever i update api_endpoint in Kobo eReader.conf, image_host, image_url_quality_template and image_url_template variables also get updated with incorrect address - it adds port :8083, but calibre-web is behind a reverse proxy and runs on standard https port - 443.

[jef]

I merged master into my PR (#1516) and built it out to test. It used the correct ports (443), but it still couldn't download either.

It also gave me a http scheme instead of https.

[Dinth]

i have just tried using internal ip address, together with http:// and 8083 port, but it still doesnt work - im getting error about my network connection

[DieterKoblenz]

Any pointers on this, I am trying to get this to work. I setup sync according to the wiki. I've changed my setup so that it uses port 8086.

http://[]en.nl:8086/kobo/308388dbf9ef860057[]/v1/initialization

Seems to give the right URLs with the correct port.

I am not getting an error, it seems to sync, but it didn't pull my books. calibre-web.log doesn't say a word. Did I miss something?

[SiemdeNijs]

Dear friends I have had been searching all the issues regarding Kobo sync, as I particularly bought a Kobo Libra H2O to sync with my wonderful calibre-web library.

I've had the same problems as others: No covers, just metadata and cannot download the epub/pdf files (filetypes ARE supported).

The issue is not resolved by changing ports or not a biggie; I use a reverse proxy to enter my own calibreweb install and now have got this to work as following:

Quick heads up: I just rolled back from latest nightly to stable, see if that helps...
Enable Kobo sync and the feature to proxy unknown requests to kobostore if unknown (made no difference this on or off for me)..
Go to your calibre-web profile and create/view token, copy only the "url" and remove the https protocol and add your calibre port as following :

Url you might get from pop-up:
api_endpoint=https://url/kobo/11111apihere11111

Now here's the easy solution I've found to get working covers and sync and downloads on my H20 Libra... I've had to remove the "https" protocol and change it to standard http, then add the port as :9696 (or whichever your install uses).

api_endpoint=http://urlblala/kobo/1111apihereofcourse11111:PORTHERE

Now go into your ./Kobo/Kobo and change the ereader conf file as following:
ONLY remove and change the URL from the standard api.kobo store , and now you replace it with your updated API url. Its important to properly DC your device and remove it. Now restart it.. and RECONNECT IT AGAIN.

Now here comes the catch: I noticed as soon as I restarted my H2O it changed the image urls and multiple other endpoints to my api url. I had to go back in the file and do a search & replace to make sure those urls did not use the HTTPS protocol and had the correct port appended.

Now the magic: DC your device properly and restart, hit the sync button and watch magic happen !

I think my problem arises mostly because I'm using a reverse nginx proxy with SSL protocol, the url my Kobo auto makes is with appended port but OVER HTTPS. This causes a protocol error and thus the e-reader sync will not connect.

I truly hope this helps other investigate this issue, for now, thank you all so much for your wonderful work on this amazing project ! 👍

Edit:
I'm now trying to find a way to keep both endpoints as calibre-web is not properly proxying requests as the feature suggest, could be a issue on my end tho but the store will not work this way.

Appending the calibre-end point underneath my kobo store api endpoint seems to work for downloading, syncing covers and books from calibre-web , but still makes the kobo store non-functional. The config file is edited as soon as the device is disconnected and it removes either one of the api_endpoints to keep only one.

Edit:
Another try was disabling my reverse proxy and connect directly over my IPV4, but even so it does not seem to proxy anything through the kobo store as it is once again not working.

Edit:
Another try was to disable SSL and even enable anonymous browsing & disable user-auth, but none seem to help.

Truly hope there's a way to still proxy the unknown requests back to the store API as else half of the functionality of the e-reader is gone when using the calibre-web endpoint (for me only?).

Edit:
Request that are proxied get the following return from Kobo, for me... :
Error CodeUnauthorizedAccessExceptionMessageUnauthorized Request. Missing authentication token.

Edit:
Seems to show only this in error logs

[2020-08-03 18:08:18,549] ERROR {cps.kobo:922} Failed to receive or parse response from Kobo's init endpoint. Falling back to un-proxied mode

[jef]

I'm going to try and follow your steps this evening and get back with success or not! Thanks for posting. I find it quite odd that the port would go at the end of the URI...

[SiemdeNijs]

I'm going to try and follow your steps this evening and get back with success or not! Thanks for posting. I find it quite odd that the port would go at the end of the URI...

In all honesty I don't think it's necessary to have the port behind the URL when using a reverse proxy and it can even bring inconveniences as not all reverse proxies handle that well. You'd basically tell the port twice, as the reverse proxy already goes toward a address:port combination. Adding an extra port behind it can cause issues like I've mentioned above with SSL protocol errors.

But that asides , thanks once again for the amazing work on the project to everyone.
Edit:
quick example: http://url:9PORTHERE9/RESTOFAPIURL

[jef]

quick example: http://url:9PORTHERE9/RESTOFAPIURL

I used this in the past and seemed to fail. Are you hosting calibre-web in a Docker container btw? Or just behind a proxy?

[SiemdeNijs]

quick example: http://url:9PORTHERE9/RESTOFAPIURL

I used this in the past and seemed to fail. Are you hosting calibre-web in a Docker container btw? Or just behind a proxy?

Not a docker install, nope. I do run it behind a reverse nginx proxy. But as mentioned above I can confirm this also works with a blunt http://IPV4:port/koboapi , as I have tried that.

It's important to change the API url to the proper formatted url and save the kobo conf file. Then, disconnect the e-reader and restart it.

It will have automatically adjusted the urls in the file corresponding to your endpoint.

Now attach the e-reader again and do a find and replace on your links. Make sure you remove the https protocol and make sure all ports/urls are correct. (In api_endooint, image cover, and image format etc). Then do a save, disconnect e-reader and voila ! Hit sync.

[pulitux]

It works like a charm for me with default port 8083. Even synchronised my shelves, as long as there's at least one new book in them. It would be a shame to lose this functionality if you opted for a special shelve to sync in #1276

[Dinth]

Now here comes the catch: I noticed as soon as I restarted my H2O it changed the image urls and multiple other endpoints to my api url.

For me it either doesnt change anything or just changes image_host, image_url_quality and image_url_template

[jef]

#1516 got merged. I will test tonight to see if this works now.

[Dinth]

Today i have tried it again on version 0.6.9.

1. Enabled "Enable Kobo Sync" together with "Proxy unknown requests" and set "Server external port" to 443 (as its running as https).
2. Restored my reader to the factory settings and replaced api_endpoint conf variable with the the following one: api_endpoint=https://calibre./kobo/238c0c41c29eb171eb06bbee7867bd1b.
   My reader still fails to sync (after running it manually it goes through "Connecting" -> "Checking for updates" > "Sync failed. Please check your Wifi and try again"). Going to the shop results in "Network error. Oops something happened to your internet connection". Before replacing api_endpoint variable Kobo shop and sync were working fine, so its not a network issue.
3. Calibre web is running on 10.10.1.12:8083 which is behind nginx HTTP proxy adding letsencrypt certificate and making it available as HTTPS on https://calibre./

When I do: telnet 10.10.1.12 8083

Trying 10.10.1.12...
Connected to 10.10.1.12.
Escape character is '^]'.
GET /kobo/238c0c41c29eb171eb06bbee7867bd1b                               

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 3
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubDomains
Date: Thu, 05 Nov 2020 13:04:03 GMT

{}
Connection closed by foreign host.

When I do: openssl s_client -connect calibre.<my_domain>:443

CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = home.<my_domain>
verify return:1
---
Certificate chain
 0 s:CN = home.<my_domain>
   i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
<my_certificate>
-----END CERTIFICATE-----
subject=CN = home.<my_domain>

issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3793 bytes and written 390 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 38194AD84F0D3621EAB29004E4F077DC7E68A8BA8C400FE70B32B3C2D4B16D46
    Session-ID-ctx: 
    Resumption PSK: 661654FB676B17E9F787C01543A0ED556D9A41D2E112B6F553A2B1DD74E1B5A9D09B1A5BBF91708E044A8CE8E75ECC8E
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - 22 9e 13 f6 e2 13 6c 11-df 33 c1 d5 ff 1a cc a0   ".....l..3......
    0010 - 9f aa 70 cd 0c 96 93 02-cf fc 9f 0f 2d d2 dc ab   ..p.........-...

    Start Time: 1604581569
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 16384
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 041953E883C7EF117058938F2B5BA5B012B0E49335050BBFD00AC3C1EA5CB0B7
    Session-ID-ctx: 
    Resumption PSK: 8F322A80FB3EA987B7FBDA75BB3E2833F9AD2BB58323C0A9B5AEC9B75EC879B5D88A6E6DD40214B294FD7878AE2E655E
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - 8d a0 b1 b5 6a f9 5c e2-83 b7 b7 07 f4 aa 28 9a   ....j.\.......(.
    0010 - 5e cc eb 93 1b a9 49 06-bb ff b2 e1 9c 2c 45 d0   ^.....I......,E.

    Start Time: 1604581569
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 16384
---
read R BLOCK
read R BLOCK
GET /kobo/238c0c41c29eb171eb06bbee7867bd1b
    <html>
        <head>
            <title>Welcome to our server</title>
            <style>
            body{
                font-family: Helvetica, Arial, sans-serif;
            }
            .message{
                width:330px;
                padding:20px 40px;
                margin:0 auto;
                background-color:#f9f9f9;
                border:1px solid #ddd;
            }
            center{
                margin:40px 0;
            }
            h1{
                font-size: 18px;
                line-height: 26px;
            }
            p{
                font-size: 12px;
            }
            </style>
        </head>
        <body>
            <div class="message">
                <h1>Welcome to our server</h1>
                <p>The website is currently being setup under this address.</p>
                <p>For help and support, please contact: <a href="me@example.com">me@example.com</a></p>
            </div>
        </body>
    </html>
closed

Although when I do open https://calibre./kobo/238c0c41c29eb171eb06bbee7867bd1b in my browser I do get a website with a single {} same as when telneting directly to the internal port

[jef]

Are you trying to get reach from a domain? Try first with your local network. I've only been able to use my local network and proxying (docker).

[Dinth]

I have changed "Server external port" to 8083, modified api_endpoint to point http://10.10.1.12:8083/kobo/f8cbf3b752f0bd5c1d5e313a1902c45d and restarted Kobo. Unfortunately, the end result is the same, I cannot sync my Kobo nor access the shop

[NotWoods]

Looking at the access logs, it seems that the Clara HD makes a GET request to /v1/products/<book-uuid>/nextread when downloading. This route isn't supported by Calibre Web. Maybe that makes a difference?

[OzzieIsaacs]

@NotWoods This is a normal behavior, there are a lot of other route not implemented, calibre-web returns an empty list in this case, so there shouldn't be a problem (at least not for me and my kobo reader)

[aniqueta]

In case it helps others, I was experiencing the same issue as described above using a calibre-web behind a Caddy HTTPS reverse proxy, with a config that includes a subdomain, a specific dir for calbire-web, and using port 443 externally. What solved API access through Caddy over HTTPS was making sure the following was in my caddyfile. Hope this helps others.

	redir /dir /dir/
	reverse_proxy /dir/* localhost:8083  {
		header_up X-Scheme {http.request.scheme}
		header_up X-Script-Name   /dir
	}

[aniqueta]

Another tip for others, for nginx reverse proxy, I needed to increase the buffer size, and Kobo sync works great. So adding the following to a typical proxy configuration should fix Kobo sync issues.

		proxy_buffer_size 128k;
		proxy_buffers 4 256k;
		proxy_busy_buffers_size 256k;

[labor4]

another solution for "no downloads" via HAproxy (problem was caused by X-Scheme) over here
cheers

[amirhomayoun]

I had the same issue and finally fixed it by adding:

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “upgrade”;
proxy_set_header   X-Forwarded-Host   $http_host;
proxy_cookie_path  /                  "/; Secure";

To my Nginx proxy manager custom location config. Then I did a "Force Full Kobo Sync" in calibre-web settings, and next sync on Kobo brought the covers and the actual books. Very happy! Thanks for the great software btw! :)

I am using the same 8083 port, and http as the scheme in NPM, using SSL with Clara HD 2.

Before adding the custom location part, I could sync my Kobo and would see the names of the books, but the images, and the actual books would not download (Clicking on the Download button would first show Pending and then Downloading, and then it would just go back to Download).

I should also add that after adding my api_endpoint to .kobo/Kobo/Kobo eReader.conf, disconnecting and connecting again, I saw my URL was added in other places in the config file (for images). I did change all of them to make sure they don't include any ports, and they all start with https.

[Dinth]

Amazing. Thanks for sharing this @amirhomayoun. I'm yet to test it but it looks exactly like a solution I've been looking for.

[MrHappy]

For HAProxy I used the following (do not know if it is overcomplete, but it works for me):

backend bk_calibre-web
        mode http
        option forwardfor
        http-request set-header X-Forwarded-Proto https if { ssl_fc }
        http-request set-header X-Scheme https if { ssl_fc }
        http-request set-header X-Script-Name /
        server calibre-web <ipaddress>:8083 check

[Microb-vn]

I use NGINX as proxy (on a synology NAS) with Calibre-web running in docker, and I needed to add these extra headers:

X-Forwarded-Proto https
X-Scheme https
X-Script-Name //

(Yes, TWO forward slashes. Using only one broke my site (infinite redirect :-@)
With these settings I can sync books, including images, and download books without any issues.

[Dinth]

So the solution proposed above worked for me for a while.
Recently i have migrated Calibre-web (and linuxserver/swag) to a new server. Even with all the configs copied, unfortunately the sync stopped working.
In fact, when i clicked Full Kobo Sync, the sync was processing "Updating books" for a considerable amount of time (and my books amount on the reader was growing to 1500+ - weirdly as i only have 1250 books in my library), but after that the sync was failing and new new books were visible on the reader.
So i have restored my Kobo to factory settings, clicked full kobo sync and attempted the sync again. This time it synced 699 books and failed - but the books stayed on the reader. I have repeated the process, but im back to the square one - the sync process is getting stuck on Updating books, the amount of books is growing, but then sync fails and the additional books are disappearing.

Obviously there's an issue with the amount of books ive got in my library (i promise, i paid the official distributor for every single one!), but maybe as a temporary workaround we could have an automatic shelf with books added in last 12 months or something like that?

PS. Also now i cannot download any of the synced ebooks, unless i change the Calibre-web settings to only sync shelves and only then ebooks previously marked for download start downloading.

[vdrover]

This worked for me.