[Feature Request] Provide a separate docker image for tachidesk "workers"

[sbordeyne]

What feature should be added to Suwayomi?

Provide a simple docker image that can be setup with a specified tachiyomi extension. The idea would be to add, in tachidesk

• configuration to specify a rabbitmq queue to use for messaging tachidesk workers in tachidesk itself
  • RABBITMQ_HOST
  • RABBITMQ_PORT
• Then, on download of a chapter for an extension, a message is sent in that queue specifying the extension id, urls to download and anything an extension might need to download the file (destination directory, for instance). Several workers could consume from the same RabbitMQ Exchange.

Why/Project's Benefit/Existing Problem

I run tachidesk on kubernetes (I can provide kustomization.yaml files for deploying it if need be), and wanted a more "high availability" setup for it. Download can be slow because they are sequentials, but with several workers, each in their own pod, I could individually scale up/down the number of worker pods depending on demand, reducing the memory limits and cpu limits I put on the tachidesk container. And speeding up the download of mangas whenever there is high demand.

I'm well aware that it is an uncommon setup (I don't think many homelabs run their software on k8s), but it would be pretty cool if it could be done (albeit I'm pretty sure it's a lot to ask, and won't ever be implemented, one can dream)

[Robonau]

downloads are only sequential on the same source, this is mainly to avoid rate limits or getting banned by the source. downloading chapters from two or more sources is done in parallel (one download per source at a time). so the whole scaling up when needed thing wont really work out.

with the DB currently being file based (database.mv.db), making the application HA doesn't really do anything to help.

[sbordeyne]

Ah alright so it would require, on top of that to be able to

• assign a proxy to each worker
• use an HA capable db (like postgres)

I think that it may be a bit too much to ask as a feature then. Too bad, I wanted to deploy a monster of manga archival on my own network :)

[sbordeyne]

But as a feature, it would also be pretty nice to be able to use "scale to zero" features of kubernetes, since if there is no downloads, there's no need for high cpu/mem limits, which on lower end hardware is a bit of a concern

[Azokul]

What feature should be added to Suwayomi?

Provide a simple docker image that can be setup with a specified tachiyomi extension. The idea would be to add, in tachidesk

• configuration to specify a rabbitmq queue to use for messaging tachidesk workers in tachidesk itself

  • RABBITMQ_HOST
  • RABBITMQ_PORT

• Then, on download of a chapter for an extension, a message is sent in that queue specifying the extension id, urls to download and anything an extension might need to download the file (destination directory, for instance). Several workers could consume from the same RabbitMQ Exchange.

Why/Project's Benefit/Existing Problem

I run tachidesk on kubernetes (I can provide kustomization.yaml files for deploying it if need be), and wanted a more "high availability" setup for it. Download can be slow because they are sequentials, but with several workers, each in their own pod, I could individually scale up/down the number of worker pods depending on demand, reducing the memory limits and cpu limits I put on the tachidesk container. And speeding up the download of mangas whenever there is high demand.

I'm well aware that it is an uncommon setup (I don't think many homelabs run their software on k8s), but it would be pretty cool if it could be done (albeit I'm pretty sure it's a lot to ask, and won't ever be implemented, one can dream)

Would you be able to provide the kustomization.yaml? I'm testing some kubernetes setups and i'm wondering if offsetting my Tachi installation to my kubernetes cluster might be a good idea.
Tho, good point, it would be cool to have the possibility to have workers dealing with scaling, and scale to zero feature is top notch

[sbordeyne]

Would you be able to provide the kustomization.yaml? I'm testing some kubernetes setups and i'm wondering if offsetting my Tachi installation to my kubernetes cluster might be a good idea. Tho, good point, it would be cool to have the possibility to have workers dealing with scaling, and scale to zero feature is top notch

Sure, you can find everything committed to git: https://github.com/sbordeyne/selfhosted/tree/master/kubernetes/komga

That repo also contains every other manifest I deployed on my homemade k8s cluster if you're ever interested. I started committing them to be able to go full gitops down the road with argocd and renovate, but haven't had the time yet to do all of that

[Azokul]

@sbordeyne Thanks a lot! I had some fun today setting up a smallish version based on your! Like the idea tho, i'm not a super huge fan of downloading stuff :')
So i went for a LoadBalanced variant without the komga part, but still a very nice repo!
Thanks again!

[taos15]

There is not need to create a specific image for the WebUI, there are some people that already did that ( I maintain my one docker image which make it easier to test stuff).search GitHub and you might find a few like mine.

[sbordeyne]

There is not need to create a specific image for the WebUI, there are some people that already did that ( I maintain my one docker image which make it easier to test stuff).search GitHub and you might find a few like mine.

I'm not sure you understood what this feature request is about. It's not about the web UI, it's about the downloading workers for tachidesk.

Having minimal workers each scoped to a specific tachiyomi extension would make having HA/scale to zero a breeze.