Calibre-web Kobo sync

[fabiosirna]

Describe the bug/problem

I've installed the docker container and connected to the calibre library.
I'm trying to sync my Kobo to calibre-web at the following address:

http://192.168.1.205:8083/kobo/{mytoken}

To Reproduce
Steps to reproduce the behavior:

1. Install docker image and configure calibre-web
2. Go to the profile page
3. Kobo Sync Token -> Create/View token
4. Browse the link (e.g. http://192.168.1.205:8083/kobo/{mytoken})

Logfile
Add content of calibre-web.log file or the relevant error, try to reproduce your problem with "debug" log-level to get more output.

Expected behavior
First I'd expect to browse a json content at the address http://192.168.1.205:8083/kobo/{mytoken}. But in my case is empty, only {} and the following:

Content-Length: 3
Content-Type: application/json
Server: TornadoServer/6.0.4
Vary: Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.5
Connection: keep-alive
DNT: 1
Host: 192.168.1.205:8083
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0

Environment (please complete the following information):

• OS: Calibre web is installed on OMV 4.0
• Python version: Python 2.7.13 (default, Aug 22 2020, 10:03:02)
• Calibre-Web version: 0.6.8
• Docker container: linuxserver/calibre-web:latest
• Special Hardware: Odroid
• Browser: Firefox latest

[dmogle]

@fabiosirna Can you try http://192.168.1.205:8083/kobo/{mytoken}/v1/library/sync. Do you get data there?

[jef]

@fabiosirna Can you try http://192.168.1.205:8083/kobo/{mytoken}/v1/library/sync. Do you get data there?

That works for me! Using the reverse proxy as well.

I ended up changing the generated URL to use https instead of http and it worked that way, otherwise I'll get a "Sync Failed. Please try again."

The interesting part is that none of the covers download and I can't download the ebook.

Here are some interesting logs I found during sync:

[2020-09-30 23:31:59,767] ERROR {cps.kobo:242} Failed to receive or parse response from Kobo's sync endpoint: Expecting value: line 1 column 1 (char 0)

[2020-09-30 23:40:57,702]  INFO {cps.kobo:913} Init
[2020-09-30 23:40:57,702] DEBUG {cps.kobo:928} Kobo: Received unproxied request, changed request port to external server port
[2020-09-30 23:40:57,702] DEBUG {cps.kobo:938} Kobo: Received unproxied request, changed request url to http://books.redacted.com:80
[2020-09-30 23:40:57,732] DEBUG {cps.kobo:851} Unimplemented User Request received: http://books.redacted.com/kobo/redacted_api_key/v1/user/profile
[2020-09-30 23:40:58,029] DEBUG {cps.kobo:851} Unimplemented User Request received: http://books.redacted.com/kobo/redacted_api_key/v1/user/loyalty/benefits
[2020-09-30 23:40:58,050] DEBUG {cps.kobo:851} Unimplemented User Request received: http://books.redacted.com/kobo/redacted_api_key/v1/analytics/gettests
[2020-09-30 23:40:58,104]  INFO {cps.kobo:130} Kobo library sync request received.
[2020-09-30 23:40:58,104] DEBUG {cps.kobo:132} Kobo: Received unproxied request, changed request port to external server port
[2020-09-30 23:40:58,231] DEBUG {cps.kobo:851} Unimplemented User Request received: http://books.redacted.com/kobo/redacted_api_key/v1/user/wishlist
[2020-09-30 23:40:58,297] DEBUG {cps.kobo:851} Unimplemented User Request received: http://books.redacted.com/kobo/redacted_api_key/v1/user/recommendations
[2020-09-30 23:40:58,402] DEBUG {cps.kobo:851} Unimplemented User Request received: http://books.redacted.com/kobo/redacted_api_key/v1/analytics/event

Another weird behavior is that if I archive the book, it'll never come back if I resync -- only new books. And I have nowhere to unarchive them (I don't think). I see now... They're archived in calibre-web. Makes sense 😄

I've also toggled on and off the "Proxy unknown requests to Kobo Store" and didn't seem to make a difference. As well as "Repair account".

---

Okay, final update here... So I decided not to go through my external network and it works with Docker.

Basically what my network looks like this is

Internet
├── https://subdomain.example.com:443
│   └── Caddy Web Server (Docker Image) 443:443
│   └── calibre-web (Docker Image) 1234:8083

Caddy has a reverse proxy setup like this:

subdomain.example.com {
	encode zstd gzip
	reverse_proxy calibre-web:8083
}

In the first case, I'm trying to hit subdomain.example.com with a forward in the calibre-web settings as 80 (also tried with 443, 1234, and 8083), but doesn't seem to work.

With connecting to my home network, I set it the api_endpoint to talk to the container directly with it's mapped port (example is 1234) and it works.

You think there is any reason as to why the forwarding isn't working? Does it have to do with the signed certs? Not really sure where to start.

Normally this is a deal breaker and won't be for the foreseeable future, but it's nice to connect a network outside of mine and download any new books.

---

One last thing (I promise)... Not all books are getting downloaded. I have < 100 books. Just found this interesting. It seems like I didn't get anymore more logs for Kobo syncing. Sort of strange. Nevermind. The books were .MOBI 😅 Just had to do a "Repair account" and it synced the rest of them that I converted.

[fabiosirna]

@fabiosirna Can you try http://192.168.1.205:8083/kobo/{mytoken}/v1/library/sync. Do you get data there?

hi @jef ,
thanks for this suggestion.
If I connect directly to this URL I got a JSON with the data of my library:

[{"NewEntitlement":{"BookEntitlement":{"Accessibility":"Full","ActivePeriod":{"From":"2020-10-01T09:02:29Z"},"Created":"101-01-01T00:00:00Z","CrossRevisionId":"12095c77-373b-42db-ac3a-71fe88721d4a","Id":"12095c77-373b-42db-ac3a-71fe88721d4a","IsHiddenFromArchive":false,"IsLocked":false,"IsRemoved":false,"LastModified":"2018-05-13T19:57:58Z","OriginCategory":"Imported","RevisionId":"12095c77-373b-42db-ac3a-71fe88721d4a","Status":"Active"},"BookMetadata":{"Categories"
...
...
and so on

I've updated the KOBOeReader/.kobo/Kobo/Kobo\ eReader.conf (MYTOKEN is the one generated by Calibre-web, omitting here):

[OneStoreServices]
account_page=https://secure.kobobooks.com/profile
account_page_rakuten=https://my.rakuten.co.jp/
annotations_host=https://annotations.kobo.com
api_endpoint=http://192.168.1.205:8083/kobo/MYTOKEN/v1/library/sync
autocomplete=https://storeapi.kobo.com/v1/products/autocomplete
book=https://storeapi.kobo.com/v1/products/books/{ProductId}
book_detail_page=https://store.kobobooks.com/{culture}/ebook/{slug}
book_detail_page_rakuten=https://books.rakuten.co.jp/rk/{crossrevisionid}
book_landing_page=https://store.kobobooks.com/ebooks
categories=https://storeapi.kobo.com/v1/categories
category=https://storeapi.kobo.com/v1/categories/{CategoryId}
category_products=https://storeapi.kobo.com/v1/categories/{CategoryId}/products
dictionary_host=https://kbdownload1-a.akamaihd.net
discovery_host=https://discovery.kobobooks.com
dropbox_link_account_poll=
eula_page="https://www.kobo.com/termsofuse?style=onestore"
featured_list=https://storeapi.kobo.com/v1/products/featured/{FeaturedListId}
featured_lists=https://storeapi.kobo.com/v1/products/featured
feedback=
fte_feedback=https://storeapi.kobo.com/v1/products/ftefeedback
image_host=http://192.168.1.205:8083
image_url_quality_template=http://192.168.1.205:8083/kobo/MYTOKEN/v1/library/sync/{ImageId}/{width}/{height}/{Quality}/isGreyscale/image.jpg
image_url_template=http://192.168.1.205:8083/kobo/MYTOKEN/v1/library/sync/fapi/{ImageId}/{width}/{height}/false/image.jpg
kobo_dropbox_link_account_enabled=
kobo_nativeborrow_enabled=True
kobo_redeem_enabled=True
kobo_subscriptions_enabled=False
kobo_superpoints_enabled=False
kobo_wishlist_enabled=True
love_data=@ByteArray(\0\0\0\0)
love_points_redemption_page="https://store.kobobooks.com/{culture}/KoboSuperPointsRedemption?productId={ProductId}"
magazine_detail_page=
magazine_landing_page=https://store.kobobooks.com/emagazines
oauth_host=https://oauth.kobo.com
password_retrieval_page=https://www.kobobooks.com/passwordretrieval.html
privacy_page="https://www.kobo.com/privacypolicy?style=onestore"
product_recommendations=https://storeapi.kobo.com/v1/products/{ProductId}/recommendations
product_reviews=https://storeapi.kobo.com/v1/products/{ProductIds}/reviews
purchase_buy_templated=https://www.kobo.com/{culture}/checkout/createpurchase/{ProductId}
registration_page="https://authorize.kobo.com/signup?returnUrl=http://store.kobobooks.com/"
review=https://storeapi.kobo.com/v1/products/reviews/{ReviewId}
review_sentiment=https://storeapi.kobo.com/v1/products/reviews/{ReviewId}/sentiment/{Sentiment}
sign_in_page="https://authorize.kobo.com/signin?returnUrl=http://store.kobobooks.com/"
social_host=https://social.kobobooks.com
store_home=www.kobo.com/{region}/{language}
store_host=store.kobobooks.com
store_search="https://store.kobobooks.com/{culture}/Search?Query={query}"
subs_landing_page=
subs_management_page=
subs_plans_page=
subs_purchase_buy_templated=
taste_profile=https://storeapi.kobo.com/v1/products/tasteprofile
user_ratings=https://storeapi.kobo.com/v1/user/ratings
user_recommendations=https://storeapi.kobo.com/v1/user/recommendations
user_reviews=https://storeapi.kobo.com/v1/user/reviews
userguide_host=https://kbdownload1-a.akamaihd.net

I'm trying to sync, but I got the sync error on my kobo. I tried also to repair the library from the device information page, restarted both calibre-web. Both calibre-web (0.6.9) and kobo firmware () are updated to the latest version (4.1.15-00416).

This is what I get in the log when I try to sync, level info is DEBUG:

[2020-10-01 09:01:55,418] DEBUG {cps.helper:682} unrar version 5.50
[2020-10-01 09:01:55,437] DEBUG {cps.config_sql:308} _ConfigSQL updating storage
[2020-10-01 09:02:29,491]  INFO {cps.kobo:130} Kobo library sync request received.
[2020-10-01 09:02:29,491] DEBUG {cps.kobo:132} Kobo: Received unproxied request, changed request port to external server port
[2020-10-01 09:03:14,501] DEBUG {cps:135} Could not parse locale "*": expected only letters, got '*'
[2020-10-01 09:03:14,536] DEBUG {cps:135} Could not parse locale "*": expected only letters, got '*'
[2020-10-01 09:03:14,665] DEBUG {cps:135} Could not parse locale "*": expected only letters, got '*'
[2020-10-01 09:03:19,575] DEBUG {cps:135} Could not parse locale "*": expected only letters, got '*'
[2020-10-01 09:03:19,653] DEBUG {cps:135} Could not parse locale "*": expected only letters, got '*'
[2020-10-01 09:03:19,690] DEBUG {cps:135} Could not parse locale "*": expected only letters, got '*'

[jef]

[2020-10-01 09:03:14,501] DEBUG {cps:135} Could not parse locale "*": expected only letters, got '*'

Weird... Looks like some language settings somewhere is incorrect? Just a guess based off error.

[OzzieIsaacs]

@jef
Could not parse locale "*" -> this is a problem with the kobo reader not sending a valid locale header, it's right nd a good sign seeing this message in the logbooks.

[OzzieIsaacs]

@jef

[2020-09-30 23:40:57,702] DEBUG {cps.kobo:928} Kobo: Received unproxied request, changed request port to external server port

-> This error messages are also good to see. the kobo reader is sending a non standard request (should be on port 8083, but id does send http://blalab.com on prt 8083)

[2020-09-30 23:40:57,732] DEBUG {cps.kobo:851} Unimplemented User Request received: http://books.redacted.com/kobo/redacted_api_key/v1/user/profile
[2020-09-30 23:40:58,029] DEBUG {cps.kobo:851} Unimplemented User Request received: http://books.redacted.com/kobo/redacted_api_key/v1/user/loyalty/benefits
[2020-09-30 23:40:58,050] DEBUG {cps.kobo:851} Unimplemented User Request received: http://books.redacted.com/kobo/redacted_api_key/v1/analytics/gettests

-> This messages are also good to see. Not all requests are implemented by Calibre-Web, especially the "social /tracking" stuff is not implemented and returning Nothing to the reader and this message to the logbook

[OzzieIsaacs]

@fabiosirna
I think the original api entrypoint you entered in the .kobo file is the right one. There is no response to this except an empty {}, this is right. The kobo reader adds the neccessary links to this entrypoint itself.

[OzzieIsaacs]

@fabiosirna
I think in 0.6.8 there have been some bugs regarding kobo sync, please try the newest version 0.6.9 (from last weekend)

[jef]

Hey @OzzieIsaacs, should I open up a different issue for the middle part of my comment earlier? Seems like it's sort of on topic, but wanted to get some feeedback.

Thanks!

[fabiosirna]

@fabiosirna
I think the original api entrypoint you entered in the .kobo file is the right one. There is no response to this except an empty {}, this is right. The kobo reader adds the neccessary links to this entrypoint itself.

Hi @OzzieIsaacs thanks for the suggestion.
I've used the original api endpoint with the 0.6.9 and it worked.

However I'm having issues. It is probably my fault because I'm not understanding how calibre-web is working :)

There was already a library synced with calibre, and I noticed that books were duplicated. So I started deleting them one eby one, but it was taken lot of time. I removed them all at once, and the idea was to sync only the ones I'm currently reading.

To make a long story short: now I'm not getting a sync error anymore, but I cannot sync the library with the kobo.

My kobo is an H20, and on the main screen has for sections:

• Home
• MyBooks
• Discover
• More

Using calibre-web I'm expecting to see my calibre library under MyBooks and download my books from there.

As a last alternative, I'm going to reset factory the kobo and see if will work again.

Edit: after investigating better the issues, I discovered that for some reason my books were archived on Calibre-web. Unarchiving is the solution to show them in the kobo library. The things is: is there a way to unarchive them all? :)

[OzzieIsaacs]

The request of syncing only a subset of books is similar to the request from #1276.
Would be good to discuss it there

[fabiosirna]

The request of syncing only a subset of books is similar to the request from #1276.
Would be good to discuss it there

At the end I made it, and the Kobo is now syncing.
I was trying to change the Archived flag from both Calibre-web and Calibre but I was not able to figure out how to do it. So i did it manually :)

I'm closing the issue.

[DHS1001]

@OzzieIsaacs

Got the same error when trying to sync Kobo and Calibre-web.
The Calibre-web, version 0.6.18, is install in a docker running in Synology Nas,
build in reverse proxy has been utilized to access via https.
Running the https://Address/kobo/Token/v1/library/sync in browser return information.
Kobo showed error "Sync failed. Please try again."

[2022-05-06 23:08:14,779] DEBUG {cps:176} Could not parse locale "*": expected only letters, got '*'
[2022-05-06 23:08:50,240] DEBUG {cps:176} Could not parse locale "*": expected only letters, got '*'
[2022-05-06 23:09:13,572] DEBUG {cps:176} Could not parse locale "*": expected only letters, got '*'
[2022-05-06 23:09:20,394] DEBUG {cps:176} Could not parse locale "*": expected only letters, got '*'
[2022-05-06 23:09:24,631] DEBUG {cps:176} Could not parse locale "*": expected only letters, got '*'
[2022-05-06 23:09:24,662] DEBUG {cps:176} Could not parse locale "*": expected only letters, got '*'
[2022-05-06 23:09:24,689] DEBUG {cps:176} Could not parse locale "*": expected only letters, got '*'
[2022-05-06 23:09:24,715] DEBUG {cps:176} Could not parse locale "*": expected only letters, got '*'
[2022-05-06 23:09:24,744] DEBUG {cps:176} Could not parse locale "*": expected only letters, got '*'

Note sure if there are other setting can try to fix this issue? Thanks

[OzzieIsaacs]

The could not parse locale message is right and expected. It‘s a problem of the kobo reader sending a header which can‘t get read by the python library

[DHS1001]

The could not parse locale message is right and expected. It‘s a problem of the kobo reader sending a header which can‘t get read by the python library

Thanks @OzzieIsaacs,

Updates:
Tried reset Kobo,
use http://192.168.x.x ip instead of the https//address.
click the "Force kobo sync"

And it sync smoothly. Thanks again