Qobj Data Layer Specification and Tracking

[jakelishman]

Last updated: 2020-08-12

This is an issue for tracking and publicising the development status of a separated data layer. This is a major breaking change to be introduced in QuTiP 5, and will change how library functions and users access the underlying data structures of Qobj. This document and issue will change over time.

Development is taking place on the dev.major branch in qutip/qutip. Pull requests implementing new features:

• Core package #1282 separating out the core QuTiP operations into a (logical but user-transparent) package qutip.core.
• Partial new data layer with basic dispatch and types CSR and Dense #1296 implementing the first parts of the data multiple dispatch and new data types.
• Replace fast_csr_matrix in core #1332 replacing the fast_csr_matrix with the new Cython type CSR all across QuTiP
• Data-layer type creation, conversion and dispatch #1338 implementing the data-layer creation, conversion and dispatcher routines.

We are writing a new "developers' guide" to explain the logic behind the design decisions, and help new QuTiP developers work on the code. You can follow initial development hosted as a repo in my user account until we decide how this will be added to the main documentation. You can view a rendered version of the documentation at https://jakelishman.github.io/qutip-devguide.

Some design documents:

• data layer
• data structure implementation choices
• original GSoC proposal (PDF)

New design elements under consideration:

• improved 'type' and 'dims' handling for tensor-product spaces

Major breaking changes to QuTiP:

• Qobj.data is no longer an instance of scipy.sparse.spmatrix, but the new abstract "data-layer type" core.data.Data
• bra * ket now returns a scalar, not a Qobj (no need to do (bra * ket).data[0, 0] or (bra * ket).tr()) any more
• 1D subspaces in Qobj.dims are now never contracted silently (they were, but inconsistently before), but the new function Qobj.contract() can be called to do this explicitly
• Qobj no longer defines __array__, and so cannot be treated as if it is a numpy array. Having this led to inconsistent behaviour when Qobj was used in arithmetic with numpy arrays from the left and right, and allowed simple code to become Qobj.dims-unsafe implicitly. To get the full array view now, you must be explicit and call Qobj.full() - it is not enough to do np.array(qobj), and this behaviour is consistent with scipy.sparse matrices.
• eseries and essolve are removed
• OpenMP is temporarily removed (but should be added back in before the release)
• most functions marked deprecated as of version 4.5.2 are removed (e.g. in qutip.correlation)
• vec2mat and mat2vec are renamed away from the terse MATLAB-compatible syntax to the more QuTiP-consistent vector_to_operator and operator_to_vector

Non-breaking changes:

• the internal file layout of the code is more organised, with physical "packages" qutip.core and qutip.solve containing the core code and the solvers respectively, but these packages export all their names to the full qutip namespace, so there are no user-facing changes
• Qobj is a bit more strict about enforcing Qobj.type-safety with input types to arithmetic operations, particularly with 1D subspaces
• Qobj understands the @ operator to mean matrix multiplication (it still understands Qobj * Qobj to be matrix multiplication as well)
• Qobj instantiation is massively faster (lower-bounded by ~5µs compared to the previous ~70µs) in internal use and when doing arithmetic. QuTiP internal functions are now much more aggressive at passing in the type and copy=False arguments, cutting out most type inference and unnecessary copying of data which were the largest time sinks.

Discussions about the implementation of multiple-dispatch methods are on-going in the repository jakelishman/qutip-dispatch. See in particular some discussion in jakelishman/qutip-dispatch#1.

Other discussions are also at the data-layer-devs discussion board.

[nathanshammah]

Thanks @jakelishman. Can we add "Qobj" in the title? Just to make very visible. Thanks!

[jakelishman]

New on 2020-06-12:

Added a new design draft about data structure implementation choices.

Added initial implementation of multiple-dispatch methods in a separate repository, which includes some discussion in jakelishman/qutip-dispatch#1.

[jakelishman]

New on 2020-06-22

Pull request #1296 containing the same dispatcher as mentioned previously, and base implementations of a Dense and CSR type, with existing QuTiP code ported to them.

[jakelishman]

New on 2020-06-27

You can follow the development of the QuTiP developers' guide in my repo: https://github.com/jakelishman/qutip-devguide, or more usefully see the rendered version on that repo's GH page: https://jakelishman.github.io/qutip-devguide.

[jakelishman]

New on 2020-07-13

Work has begun on porting the Qobj internals to use the new data structures. As part of this, I am particularly concerned with speeding up Qobj instantiation in the base mathematical operations. A lot of the slowdown comes from internal handling of the dims and type parameters, and there are some suggestions for how these can be improved in #1320.