Using qutip_qip as an optional package for qutip

[BoxiLi]

Background

This is an attempt to set up qutip_qip to be used as an optional package for qutip when installed. It can also be generalized to other qutip family packages such as control, lattice or tensorflow. As there are a few downstream packages using those modules (especially control), allowing this will make the upgrade to qutip 5.0 easier for them.

Expected behaviour

• qip will be a subpackage for qutip but it is not imported to qutip in the top-level __init__.py. This is one of the original goals: make qutip importation lighter.
• If only qutip is installed, importing qutip.qip will pop an ImportError with instruction for installing qutip_qip. In principle, it also allows a completely different version of qutip.qip be implemented in qutip.
• If qutip_qip is installed, importing qutip.qip will automatically using the qutip_qip library, with no difference in the user interface.
• optional: automatically install qutip_qip if some flag is given when installing qutip. Haven't look into this yet.

Implementation

Using entry_point, it turned out to be easier than I thought.

In qutip_qip, add a parameter to the setup function:

setup(
      ...
      entry_points={'qutip_qip_plugins': [
                'qip_module = qutip_qip',
            ],
      },
      ...
)

The "group id", qutip_qip_plugins, will be used in qutip to identify this entry point group. The string qutip_qip is the module to be exported to qutip, which can also be replaced by a submodule such as qutip_qip.compiler.

In qutip, write the following in qutip/qip/__init__py

from importlib import metadata
import sys
try:
    for qutip_qip_entry in metadata.entry_points()['qutip_qip_plugins']:
        qutip_qip_module = qutip_qip_entry.load()
        sys.modules["qutip.qip"] = qutip_qip_module
except KeyError:
    pass

This will look for the entry point group id, load qutip_qip if it is found. Then we point the name qutip.qip to the module qutip_qip in sys.modules.
With this, everything works smoothly

import qutip.qip as quqip

from qutip.qip.circuit import QubitCircuit

import qutip.qip.circuit as circuit
circuit.QubitCircuit

All of them return/use modules pointing to qutip_qip

Note

I'm a bit surprised that even import qutip.qip as quqip gives me a module pointing to qutip_qip, because the sys.modules is actually changed from qutip.qip to qutip_qip during the importation. Then I found that it is explained here in the footnote. The import function does not use return, rather it looks the name up in sys.modules and gives that back. However, it is also mentioned that "This is implementation-specific behavior that is not guaranteed to work in other Python implementations." So maybe we need alternatives if we want to go beyond CPython.

Move it to qutip/__init__.py may solve this since qip will always be loaded when initializing qutip. It is not a very bad option because it will only be loaded if qutip_qip is installed. But still this might not be an optimal choice.

[hodgestar]

My instinct would be to opt for a simpler solution use extras_require:

setup(
    ...
    extras_require={
        "qip": ["qutip_qip"],
        "control": ["qutip_control"],
        "all": ["qutip_qip", "qutip_control"],
    },
    ...
)

The entry points solution will still require qutip_qip to be already be installed.

The optional import then becomes just:

import sys
try:
    import qutip_qip
    sys.modules["qutip.qip"] = qutip_qip
    del qutip_qip
except ImportError:
    pass

Thoughts?

[BoxiLi]

Thanks @hodgestar! Indeed extras_require could be the solution for optional installation, e.g. using pip install qutip[qip]. I haven't started to try optional installation but it should work.

It seems to me that entry_point and the optional import in your second code snippet should be equivalent. They both aim to rewrite the path of qutip.qip. And they can exist in parallel with extra_require, which deals with optional installation. Is that true?
Then I'm a bit confused, why people still use entry_point to develop extensions to a package if it can be done so easily with normal importation (I know it can be used to create console script)? Did I miss anything? Or is it just an overkill here.

[hodgestar]

It seems to me that entry_point and the optional import in your second code snippet should be equivalent. They both aim to rewrite the path of qutip.qip. And they can exist in parallel with extra_require, which deals with optional installation. Is that true?

Correct.

Then I'm a bit confused, why people still use entry_point to develop extensions to a package if it can be done so easily with normal importation (I know it can be used to create console script)? Did I miss anything? Or is it just an overkill here.

entry_point is useful when there might be many plugins and one doesn't know what modules or functions will provide the functionality. In this case we already know that we want to check whether qutip_qip is installed, so entry points are overkill.

As an example of entry_point usage, one could imagine a future where qutip provided an entry point for solvers and then third parties could supply solvers for qutip and we wouldn't have to update qutip itself at all. Perhaps this is even something we should do at some point, but let's wait until we get there.

Small aside: With entry_points its more common to do, e.g., entry_points(group="qutip_solvers"), than entry_points()["qutip_solvers"] because it takes care of the case where there are no entry points installed for the group nicely. See https://packaging.python.org/guides/creating-and-discovering-plugins/#using-package-metadata (apologies if you read that already).

[BoxiLi]

Thanks for the expanation! Now I get it. So I will keep this discussion as an example if we ever want to use entry_point. For qip we can probably go for the simple solution.

With entry_points its more common to do, e.g., entry_points(group="qutip_solvers"), than entry_points()["qutip_solvers"] because it takes care of the case where there are no entry points installed for the group nicely.

Actually entry_points(group="qutip_solvers") throws an error for me (Python 3.9). And if I look at the docs for setuptools it is using entry_points()["qutip_solvers"]. I suspect there are different versions?

[hodgestar]

Actually entry_points(group="qutip_solvers") throws an error for me (Python 3.9). And if I look at the docs for setuptools it is using entry_points()["qutip_solvers"]. I suspect there are different versions?

Yep -- the example I posted only works in Python 3.10 onwards. For earlier Python's one can pip install importlib_metadata and use importlib_metadata.entry_points instead.

[BoxiLi]

Ah I missed that line sys.version_info < (3, 10). Didn't expect it to be that new.

[hodgestar]

P.S. Regarding import foo first looking for sys.modules["foo"] -- it's hard to imagine any Python implementation not having this optimization, so we're pretty safe depending on it. A lot of Python software relies on this in one way or another, and searching again for modules that have already been loaded every time import is called would be very slow.

[BoxiLi]

The key point is not that import foo first looking for sys.modules["foo"]. I was pointing to the following footnote:

The importlib implementation avoids using the return value directly. Instead, it gets the module object by looking the module name up in sys.modules. The indirect effect of this is that an imported module may replace itself in sys.modules. This is implementation-specific behavior that is not guaranteed to work in other Python implementations.

I'm imaging what happens is the following:
When I do import qutip.qip as quqip, Python will first load qutip, then qutip.qip. At the point that it is running qutip.qip/__init__.py, there should already be a module for qutip.qip created, i.e. sys.moduls["qutip.qip"]=module("qutip/qutip/qip/__init__.py").
But during this process (running qutip.qip/__init__.py), we change this object to
sys.moduls["qutip.qip"]=module("qutip_qip/__init__.py").

If I understand that footnote correctly, it indicates that in some implementation, import will still return the old module object with the path pointing to "qutip/qutip/qip/__init__.py"?

If that is true, then running
import qutip.qip as quqip
will give me a module pointing to qutip/qutip/qip/__init__.py, not qutip_qip/__init__.py

[hodgestar]

@boxli Ah, I see. I also think its unlikely that another Python implementation would do something different in that case too. It's fundamental to how a Python module is loaded that there are two steps -- 1) create the module object and 2) run the contents of the module. In between the two steps, the module from step 1 needs to be placed in sys.modules to avoid infinite recursion and step 2 runs arbitrary Python code, so it can do anything including update sys.modules. Returning a module that wasn't what ended up in sys.modules would violate many sane assumptions (e.g. that import foo as x; import foo as y would result in x is y).

I think we fine to just try the simple way and if it breaks on some Python implementation someday we can fix it.