Redesign Nova2/3 search engine

[depau]

I wanted to make qBittorrent's search engine standalone, so I took the code and tried to make it more independent.

As it changes many things I'm not opening a pull request immediately, I just wanted to show you the code first to see if you like it.

https://github.com/Depaulicious/Nova6

What's different from your code:

• Supports both Python 2 and 3 out of the box: I patched it to use requests instead of urllib*. I renamed it to nova6 because 2*3=6, like six
• I patched it to be used with setuptools. nova6.py and nova6dl.py shouldn't be called directly anymore: setuptools will set up two executables in PATH that will pull the required functions in and make it work the same way it used to.
• It uses argparse instead of just reading sys.argv, so that more command line arguments can be added.
• Search engines don't have to be placed in nova*/engines: this behavior is still supported for backwards-compatibility. Engines lookup directories should be specified in command line with the -d argument.
• If -p command line argument is provided, it will output some progress information after an engine is done searching.
• The library is now a full-fledged package. While qBittorrent's nova is technically a package because of the presence of __init__.py, it uses same-directory imports instead of same-package imports, making it work like a script pulling in some other scripts.
• This caused some issues with plugin compatibility, which is why I had to add a few (very very ugly, I know) sys.modules aliasing hacks. They make sure that modules can still import package modules the way they always used to. It also aliases some commonly used modules that have been moved/renamed in Python3 to their old name.
• All this changes make it incompatible with multiprocessing.Pool, because multiprocessing doesn't really know how to reconstruct this environment in a separate process. I switched to multiprocessing.dummy.Pool (a threading.Thread-based pool) to keep it in the same process. The performance loss is not noticeable: blocking is mostly due to network operations only, which are done by the operating system. They run in parallel regardless of GIL.
  A Process based tool would also make it a bit complicated to display progress.

Let me know if you like it, I'll be glad to add some more changes and help you merge it into qBittorrent.

P.S. why I did this? I'm making a standalone web-based torrent search interface compatible with qBittorrent's search plugins. I'm still working on it because ­— you know — web browsers, I'll upload the code to GitHub very soon. nova6 seems very reliable and it's shaping up very well.

[depau]

Since Python3 is the current version, should you not alias Python2 modules under their newer names instead?

I detect Python version at runtime and alias the modules accordingly. For example, if module m2 in Python2 has been renamed to m3, m3 will also be available as m2 in Python3, and m2 will also be available as m3 in Python2.

@Depaulicious: There has been quite a bit of interest in doing exactly this, so a pull request would not be unwelcome.

Keep in mind that devs on this project are fairly hard on PR authors, so you will most likely need to make many changes before this can be merged. ;}

I figured 😂 That's why I asked here first, I'd like to know what's right and what's wrong, then eventually I'll make changes and PR.

P.S. my Nova+Flask-based search engine is taking shape 😉

[depau]

Speaking of that, you should probably rename this issue for better visibility

Doing it now :)

I would recommend reverting this one line back to name='novasearch',.

I see you commented something somewhere in the huge commit, but I can't find the comments. Would you please post here the links to the lines or email them to me?

Thanks!

[sledgehammer999]

1. Based on the info in the bullet points, I am ok with it. Although I don't understand the last one. (I am no python dev)
2. Since I am no python-dev I didn't even try to glance at your new code.
3. Maybe name it something else instead of six. Like novaX. Theoritically speaking, when we reach python6 what would this lib be named?
4. If you keep compatibility with current API specs the I assume it is pretty safe and reasonable to open a PR. Then the right people will see it and start reviewing.

[depau]

@sledgehammer999

1. Based on the info in the bullet points, I am ok with it. Although I don't understand the last one. (I am no python dev)

What multiprocessing.Process does (optionally on Unix, always on Windows) is spawn a new interpreter, re-import all needed modules, pickle any data needed by Process, find itself, then run itself.

In order to make nova a proper package, thus setuptools-compatible, I had to change the way plugins are imported. In my implementation it is not clear for Process where to find the plugins because they're not in the same directory as nova2.py any more.

Currently nova, even though it looks like a package, does not work like one because qBittorrent calls one of its scripts directly. For this reason plugins can (in theory) be imported simply with import engines.plugin_name, knowing the name of each Python file. In my nova there is a function that picks the right import method for the current Python version and, given a path to a Python file and a module name, returns the imported module. This kind of import can't be reduced to a basic import statement, so multiprocessing really know how to deal with it.

This behavior also breaks my module-aliasing feature (which is required to maintain API compatibility with existing plugins), as sys.modules changes are not picked up by multiprocessing (and that's a good thing, if you were wondering).

This isn't an issue on Unix, technically, because we have fork (see this). However it turns out the forkless behavior is the default for Pool. I didn't try to force-enable it because it wouldn't work on Windows anyway, it would just add unnecessary code to maintain.

A thread-based pool is a good compromise: we have a good degree of concurrency (network operations run concurrently for sure) and we get to keep the environment without adding even nastier hacks to the code.

3. Maybe name it something else instead of six. Like novaX. Theoritically speaking, when we reach python6 what would this lib be named?

Well I think well have to wait a little bit before Python6, and it's not a race or something anyway 😂 As I said, I liked how six developers came up with the name. If you guys like novaX (which would have to be novax because of Python naming conventions) or come up with a better name, awesome! We can change it.

I personally don't really like novax, I think it sounds like a medicine name, a laxative 😂

4. If you keep compatibility with current API specs the I assume it is pretty safe and reasonable to open a PR. Then the right people will see it and start reviewing.

Fine, I'll open it soon then. Beware that it would still need some changes in qBittorrent's code because

1. It should be installed and used with setuptools, even though it may work without it
2. Engines should not (and cannot, if installed with setuptools) be placed in the package's directory. This means we need to add the -dargument and place the plugins elsewhere (not a very big deal IMO).

It's a minor change, but I'd rather have someone else do it or help me do it: I don't have a lot of experience with C++ and have never seen qBittorrent's code.

[depau]

@LordNyriox

My apologies. In your initial refactoring commit, you changed the name field in (current) Line 9 of
[https://github.com/Depaulicious/Nova6/blob/master/setup.py].

Since you used the name novasearch when you initially created the setup.py file, I felt that the change from setup( name='novasearch' to setup( name='nova6' in that line was most likely unintentional.

I foolishly deleted the comment in one of my over-zealous clean-the-clock moods. ;}

Okay 😂
I initially used novasearch, then thought nova6 was a better name and renamed everything. I also changed a lot of things in that file. I had my IDE auto-generate it and didn't realize it did many things wrong (like not using setuptools 🤦)

[depau]

One more detail I think is somewhat important:

in setup.py I put my name and email in the "author" fields. I'm obviously not the author, but, not finding any other fields in PyPI documentation, I interpreted it as the current maintainer.

Let me know if you think it's okay.

[sledgehammer999]

Thx for info. Unfortunately I can't tell if yours is the right direction about multiprocess/threading.
About the qbt bundling and setuptools. What's the benefit of using setuptools from the qbt standpoint?
As for the plugins installation point: If the plugins are backwards-compatible it doesn't matter where we install them. The old client versions will work with their nova, while the new client versions with the new nova6 and the new plugin's save path.

[depau]

Thx for info. Unfortunately I can't tell if yours is the right direction about multiprocess/threading.

I guess I'll wait for more feedback.

About the qbt bundling and setuptools. What's the benefit of using setuptools from the qbt standpoint?

For qBittorrent, absolutely nothing. It could just continue to be a "fake package" and it would still work. Using setuptools makes it easy to separate its development from qBittorrent's and make it standalone. It would work just like any other Python module that is available in PyPI or distro repos.

Setuptools helps making sure dependencies are present, build proper executable scripts that run code from the package and allows easy publishing to PyPI.

As for the plugins installation point: If the plugins are backwards-compatible it doesn't matter where we install them. The old client versions will work with their nova, while the new client versions with the new nova6 and the new plugin's save path.

That is correct.

[Chocobo1]

I patched it to use requests instead of urllib*.

You also mentioned setuptools, does it mean the user has to install these 2 packages if he wants to use search? (either qbt bundles it or user installs it)?

Although you listed the features in first post (I assume which are good for your project), from qbt standpoint, may I ask what are the benefits? (aside from py2 & 3 unification).

[DoumanAsh]

I'd be interested to see how integration will look exactly.
The important thing is that we need nova being independent of user's environment.
As long as user has python, it should just work, do you intend to address it?

@sledgehammer999
multiprocessing.Pool is not really a serious change, aside from potential minor degradation of search speed with big number searches.

P.s. please ping me so that i could review your PR. I like idea of unifying py2/py3 code base

[sledgehammer999]

I'd be interested to see how integration will look exactly.
The important thing is that we need nova being independent of user's environment.
As long as user has python, it should just work, do you intend to address it?

Ideally, we just change the cmd line we use to call python+script. The results should be in the same format as now.

For qBittorrent, absolutely nothing. It could just continue to be a "fake package" and it would still work. Using setuptools makes it easy to separate its development from qBittorrent's and make it standalone. It would work just like any other Python module that is available in PyPI or distro repos.

Setuptools helps making sure dependencies are present, build proper executable scripts that run code from the package and allows easy publishing to PyPI.

If this plans goes through, then IMO it is time to choose a more descriptive name than "nova". Anyone looking at the package in PyPI should have an idea on what it is by its name.
(just throwing ideas).

Although you listed the features in first post (I assume which are good for your project), from qbt standpoint, may I ask what are the benefits? (aside from py2 & 3 unification).

+1

I patched it to use requests instead of urllib*.

You also mentioned setuptools, does it mean the user has to install these 2 packages if he wants to use search? (either qbt bundles it or user installs it)?

I suppose both of those packages are available if user has python installed. @Depaulicious right?

[DoumanAsh]

I suppose both of those packages are available if user has python installed. @Depaulicious right?

As far as i know requests is not part of standard distribution.
And the way how @Depaulicious is intending to address it the most important issue of this proposal

[depau]

@sledgehammer999

Ideally, we just change the cmd line we use to call python+script. The results should be in the same format as now.

Yep. I barely touched the result-printing stuff (only addressed some 2/3 compatibility issues).

I suppose both of those packages are available if user has python installed. @Depaulicious right?

TL;DR: on Linux: they're likely already installed. On Windows: setuptools is there, setuptools downloads requests.

Long answer:

requests is not part of the standard library. Technically setuptools isn't either, but Python installations come with easy_install, which is based on setuptools. easy_install can pull in requests from PyPI.

In any case, you should note that, in 2017, most Python modules that need to connect to HTTP have already switched to requests because of it's convenience, so it is installed by default in many Linux distros (for example, Ubuntu - grep for python3-requests).

---

@Chocobo1

You also mentioned setuptools, does it mean the user has to install these 2 packages if he wants to use search? (either qbt bundles it or user installs it)?

TL;DR: Yes.

Long answer:

there are two proper ways to install a Python package, and that depends on whether you want to have it installed in a local, private environment, or system-wide.

In the former case you would need to create a virtual environment in some private user-writable program directory (i.e. .local/share/qBittorrent), then run pip install nova6 or easy_install nova6 in it, which will also pull in requests. This means qBittorrent will have to depend on virtualenv (and python, of course), everything else will be downloaded automatically.

This method has the advantage that qBittorrent will have the ability to automatically update nova by simply running pip install --upgrade nova6 inside the virtualenv.

In the latter case you'd need to prepare everything at build time, so it would only depend on requests.

In my opinion, while the latter would work very well on platforms where a proper packaging system is available (on Linux you'd just make a package for nova and depend on it, other dependencies would be pulled in by the package manager), the former case is more "universal".

Although you listed the features in first post (I assume which are good for your project), from qbt standpoint, may I ask what are the benefits? (aside from py2 & 3 unification).

As I said earlier, it would make it easier to automatically update the search engine when needed, and have the search engine in a separate repository.

---

@DoumanAsh

The important thing is that we need nova being independent of user's environment.
As long as user has python, it should just work, do you intend to address it?

Of course. That's one of the points of using setuptools. A virtualenv would make it completely independent of the rest of the system.

[depau]

Screenshot of the installation of nova6 on Windows, on a freshly installed Python3:

python3 -m venv venv creates a virtual environment in the venv directory. venv is part of Python3 standard library, it's available for download on Python2.

Entry points for modules installed in virtual environments are found in venv/bin on Unix and venv\Scripts on Windows.

So to automatically install nova, that's all you need to do. Then you just need to download the plugins somewhere in the filesystem and run it.

[sledgehammer999]

Personally, and speaking from the qbt POV, I am not interested in making it more complex to deploy the search feature.
python 2/3 unification is sweet though.

[Chocobo1]

TL;DR: on Linux: they're likely already installed

Not true in my case, my arch linux doesn't install it by default.

In any case, you should note that, in 2017, most Python modules that need to connect to HTTP have already switched to requests because of it's convenience

Agree, but this is a C++ project with some python scripts bundled.

Long answer:
there are two proper ways to install a Python package

Thank you for the lengthy explanation.

As I said earlier, it would make it easier to automatically update the search engine when needed, and have the search engine in a separate repository.

IIRC, there are C++ parts that interact with it, so it's not preferred to update python part alone IMO, then it nullifies the need a of separate repo.

Personally, and speaking from the qbt POV, I am not interested in making it more complex to deploy the search feature.

+1

[DoumanAsh]

@Depaulicious I don't think most users would even bother accessing python enviornment and manually update engines/nova.
Everything should be implicitly handled by qBt.

Personally i just use portable python 3 without nay virtual env, and i don't need multiple enviornments :)
Albeit I have requests and setuptools installed, i'd prefer that qBt wouldn't make any assumptions about it

Anyway, I'll review your PR so ping me and will see how it looks exactly

[depau]

So, to recap:

1. You don't like it being too separate from qBittorrent
2. You still like the 2/3 unification thing

How would you like it if we still keep it in a separate repository, still keep setuptools support for standalone usage but I make two wrappers that qBittorrent can call in order to use it without actually installing it anywhere?

It can be included back in qBittorrent's repository using git submodule, which also has the benefit of referencing a specific commit, therefore having a specific revision for any particular version of qBittorrent.

It would still depend on requests, of course, which can be made a qBittorrent dependency on Linux distros and can be installed during Python installation on Windows (just run python.exe -m pip install requests after Python has been installed).

[depau]

I finally opened the pull request (#7496).
Let me know what you think about it.