-
Notifications
You must be signed in to change notification settings - Fork 462
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Discusion: Breaking changes in qBittorrent search-engine #121
Comments
👍 |
I like it, but I'm already a Jackett user. |
Updated the first message with the changes required. @Chocobo1 Would you like to work in the development? I can test everything carefully. :) |
I probably won't have much time to do anything significant. |
Of course you can. You have all the details in the first message. Just mention me when you have a PR or something. If at any point you are not longer interested in implementing this, please, notice me. |
Imho the python integration has been there for a long time and it would not be ideal to drop it completely after all the work it required, just drop python2. Also, not all sites are supported and you can't use the "go to the description page" option. |
These would be the sites that we have but jackett does not:
But yeah, it sounds like a sensible idea, and there is no reason why these sites above could not be implemented in jackett.
Jackett does support that |
I opened some requests in Jackett to add them. |
Has anything already been done regarding this? I did upgrade my qBittorrent to 4.2.3 after which I still see all the plugins I had before, but "categories" only list "All categories". Also, sometimes plugins do not find anything anymore. I agree that Jackett is a powerful plugin, but currently setting it up, running a separate process/server, configuring API keys and it being much slower than "normal" plugins, is not that good. I also see many errors in Jackett log, which implies that not everything works as reliably (of course, I have most of the indexers enabled for testing purposes, which might cause problems too - I will try to limit indexers). Search functionality has been a killer feature for me so far when choosing a torrent client and if this changes then it needs to be easy/transparent to use (and Jackett is definitely not). |
As far as I know no one is working in this. Most of the plugins included in qBittorrent are working (just make sure you have them updated). The categories are not tested well, just use "all categories".
I'm Jackett developer too. These issues are real and are caused because Jackett has an API called "agregator" that sends the requests to all trackers at the same time.That API is not designed for use in production and its use is discouraged. It doesn't work well because it will wait until all trackers respond and if one tracker fails it causes problems in the others. In the current implementation of qBittorrent we are using that, but with the proposed changes the idea will be to list all indexers in qBittorrent and make separate requests. That is working really well in Jackett.
I agree but qBittorent developers are not interested in maintaining this (an most of them use jackett anyway). In addition, it is increasingly difficult to maintain trackers because they have security measures such as re-captcha and cloudflare. Jackett has mechanisms to circumvent those protections. |
@ngosang thank you for your thoughts.
My point was that they were working before and stopped working after upgrading. That's why I asked. It doesn't really matter to me because I always have been using "all categories" anyway. Just an observation.
Looking at the Jackett logs that was my initial suspicion too, that it works exactly like you just described in here. Good that you confirmed it, since that explains a lot. When I did enable all (public) indexers in the beginning then I had to start looking at the log and disable some, which were failing thus causing entire search to not return any results. Using your proposed API does make sense and sounds like a good idea.
I totally understand this. One way of solving this problem would be to create better integration between Jackett and qBittorrent where Jackett is bundled with qBittorrent or can be installed somehow throught qBittorrent and it would start/stop Jackett server and configure API key automatically or whatever so that end-user would not need to know anything about Jackett or how to configure it etc. I'm quite sure that this is also not a small task and I totally understand that PRs are welcome since I'm developing some OSS stuff too. Just my thoughts how things could be improved. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
@ngosang |
It says that I can optionally get data in the form of JSON. Does it really work in Jackett? |
😅 You're right of course :) Good thing this won't be an issue then.
Again, a little bit of confusion from my part. Seems we just need to make a few requests using Jackett's API, i.e., we only ever act as a client, which is even simpler.
My understanding (and hope) is that the maintenance burden will be minimal for us. We "just" have to implement the required API calls, and process the received data and display it nicely in the UI. In fact, since apparently Torznab/Jackett's API is actually a "de facto" specification/protocol at this point, not just some internal one-off thing from some very particular 3rd party program, we could even think about ditching the search plugin concept entirely and just say qBittorrent implements support for the specification as a self-contained subsystem that is part of the core (I think this is exactly what @ngosang is suggesting, just worded a little differently) - i.e., we accept data from any program that sends it in the expected format, conformant to the spec. This would be much like the relationship that qbittorrent has with RSS - qbittorrent implements the necessary bits of the protocol/spec so that the user can add any valid feed; there is no community-maintained repository of "RSS plugins" for each individual site. Of course, unlike the Torznab/Jacket API, I beleive RSS is standardized with an actual RFC (or multiple?), which is a stronger guarantee, but I believe the point still stands in spite of that. |
This is one of the key aspects of my concept as well.
I insist that we are no longer responsible for installing any third-party applications. If we do support Jackett, the responsibility for installing and configuring it should be entirely on the user himself. All qBittorrent should care about is maintaining the parameters it needs to interact with Jackett. Offtopic:
I hope that Torznab/Jackett's API is something more strict/reliable than the RSS. Personally, I can't bring myself to call RSS a standard. It looks like a bunch of "may", "recommended" , etc. In addition, even what is defined there more or less strictly, is not respected by most of the so-called "web developers" who implement it on their sites. Because of this, our "generic" RSS support has a bunch of issues with supporting certain feeds. Really, I'm not so strictly against implementing Jackett support instead of the existing subsystem. This looks like a worthwhile compromise. I am waiting for answers to my comments/questions above from interested parties (especially from @ngosang). Perhaps I will do this job if we agree on the key aspects of the problem. |
No. You only need to make HTTP requests to a remote service and parse the response. Jackett is stateless so you don't have to keep sessions or other things between requests neither. It's really simple.
The response is standard XML and can be parsed with Qt. The specification mentions JSON too but it's not implemented in Jackett. I never saw other Torznab programs using JSON, always XML.
Don't worry about that. As I said, I'm maintainer and we have more than 20k daily downloads. There are other projects like Cardinann that follow the same Torznab specs. Jackett is not going anywhere, too many projects depend on it.
It provides that information and more => https://github.com/qbittorrent/search-plugins/blob/master/nova3/engines/jackett.py#L136
Yes and no. The basic functionality is covered by the plugin. Search request + XML response parsing. @glassez I elaborated a document with the details. Take your time. |
It seems that the concept of using Torznab differs from mine only in the protocol, doesn't it? @ngosang |
Cheers for the document, really nice. I have 2 alternative proposals concerning "indexer management": Is it possible to obtain a list of all possible indexers by querying the API? If so, qBittorrent could automatically make such a request and show all available indexers to the user. This also allows auto-filling all of the 4 mandatory text fields except the API key (page 4, 2nd figure), assuming the API returns a usable Maybe indexer management could be a tab instead of a dialog? Not a very important decision, but worth thinking about. Also, you mention that querying all indexers is error-prone and incurs a performance penalty. We should probably warn the user (with a red-lettered label or something) that executing searches with many indexers enabled may take a long time. Another note: It should be possible to search with only a subset of enabled indexers, not just one or "all enabled" (but that capability can probably be left out of a first implementation). |
Well, I'm going to do this job in the meantime.
In any case, it will initially be a dialog, since the other option is controversial.
I still insist on implementing it to be configurable at "independent indexers" basis I still insist that it can be configured based on "independent indexers", so we don't have a hard dependency on the Jackett or anything else. Just a set of individual endpoints that support Torznab protocol. |
Of course, I don't mind too much if you do end up implementing it as a dialog, this is simply a suggestion. I agree with the rest of your post. |
Well, it looks acceptable, except that it loses a bit of vertical space, which might worry someone. |
Good point, agree that horizontal space is cheaper. But the button to bring out the dialog would take up approximately the same amount of space. So I guess the ideal solution would be to position the tabs vertically on the right, the same we do it for the banned IPs tab. |
Nice to hear that. Please, quote me when there is something to test. I can help with that. |
In the docs above is said:
But actually I see Jackett provide |
Yes, you are right. I will fix it in Jackett today. How it affects you? |
👍
I needed to know what format the data is provided in Torznab in order to parse it. When I saw I have a sad experience of working with RSS, when even very soft requirements of the standard are often not met by server side. So in this situation, I would prefer that the Jackett bugs, if found, be fixed in Jackett without any workarounds in qBittorrent. |
@glassez it's already fixed. New Jackett version will be released tomorrow. If you find more things just tell me. Released https://github.com/Jackett/Jackett/releases/tag/v0.18.255 |
This comment has been minimized.
This comment has been minimized.
Well, I've finally completed the bulk of the work on this (qbittorrent/qBittorrent#15126). It's still a draft, since some little things are missing, such as validating the input data, etc. But you can already try it in practice. At the same time, I would have heard preliminary feedback. Just please don't ask for anything more advanced than what is there. I would like to limit myself to the basic functionality at the first stage. Advanced features can be added later. |
I repeat qbittorrent/qBittorrent#15126 (comment) here. Maybe I'll get an answer after all...
|
Hello all, What is the state of the proposed changes on macOS version of QBT 4.5.0 / LT2.0? According to the WIKI page for Search Plugins this has already been implemented yet I am seeing a nova3 directory wasn't removed upon install of v4.5.0 and I still see Python plugins. No mention of adding a torznab indexer. Do we need to remove the nova3 directory for the new torznab feature to be implemented properly? |
It's not implemented yet and nobody is working on it. The development is here => qbittorrent/qBittorrent#15126 |
@ngosang I had some issues with Jackett (since resolved) and that led me to discover Prowlarr. I saw that Torznab implementation has stalled, but since Jackett already works I thought that means qBt can talk Torznab well enough already and I could just feed it Prowlarr's info in jackett.json since it's just another Torznab implementation. |
@amamelia Jacket is working fine. The qBittorrent plugin only works with Jackett but it could be modified to work with Prowlarr. |
Update: Proposal => qBittorrent.pdf
Hello everybody,
TLDR: I don't have time to maintain the plugins, the search-engine code needs a huge refactor. I'm proposing to remove all search plugins and Python code and make a native integration with https://github.com/Jackett/Jackett
First of all, I think this feature is used by thousands (take a look at Reddit) and qBittorrent is the only bittorrent client with this functionality. So, I don't want to remove it. Furthermore, in the latest releases is supported in the WebUI making it convenient for seedboxes.
The current implementation (code) is a mess, both, the C++ and Python parts. In this issue #84 I list all required changes to make it clean and more maintainable. That are a lot of changes, breaks compatibility, I don't have the skills to make the changes in the C++ code, and more important, there aren't maintainers for the plugins.
Instead of doing the changes, I propose to remove all Python code, including official and unofficial plugins and make native integration with Jackett for the following reasons.
I want to know what do you think about and I would like you to try Jackett https://github.com/qbittorrent/search-plugins/wiki/How-to-configure-Jackett-plugin
ping @qbittorrent/demigods @qbittorrent/webdev @qbittorrent/frequent-contributors @qbittorrent/bug-handlers
UPDATE: Implementation draft
NOTE: In the first version I will try to make as few changes as possible.
1. Enable search engine
2. Install plugins / search configuration
3. Search UI
4. Perform searches with Torznab
** qBittorrent column => Torznab
** name => title
** size => size (in bytes)
** seeders => seeders
** leechers => peers - seeders
** search engine => torznab indexer name configured by the user (the real name of the indexer isin the response but it can confuse the user)
** description page => comments
** download link => link (can be a maget or a http link)
5. Future steps
The text was updated successfully, but these errors were encountered: