I'll be taking over core development of vector

[jpivarski]

Hi watchers and @nsmith-, @jrueb, @marinang, and anyone else who has been interested in vectorized vectors!

A little over a month ago, @henryiii and I worked out a plan in which he would become in charge of the scikit-hep/awkward-1.0 build and deployment and I would bring the scikit-hep/vector package up to 1.0 status. I said I'd start that in March, and managed to get to it in time, as it's now March 2. I think this will take about a month (and then Uproot 4 file-writing is probably next).

The default branch for this project has already been set to develop, rather than main. I'll be doing this development in PRs that target develop.

I'm going to start by clearing out src/vector, tests, and the documentation files in docs, leaving the structure of everything else as it is: package configuration, continuous integration, the structure of the documentation, etc. This is to "get my head around" what is done and what isn't done—I'll be looking at the old code, and the result will have largely the same structure. But, for instance, I'll be turning on the type safety of the protocols (which ensure that we haven't missed any methods) at the end of development, rather than at the beginning.

One structural change that I do plan to make is to make the NumPy implementation a subclass of np.ndarray again (as it was in hepvector). That forces the components to be contiguous in memory, which is a sometimes-desirable feature. Awkward Array handles the case when it isn't desired.

I'll be looking closely at all of the other Lorentz vector projects for structure and naming conventions:

• The main branch of this repo and henryiii/hepvector.
• ROOT's TLorentzVector and especially Math::LorentzVector, as the latter handles multiple coordinate systems as we want here.
• scikit-hep/math, where we specifically thought through what the Scikit-HEP Lorentz vector interface should look like (3 years ago). That's closely related to the uproot-methods TLorentzVector class—one was made to mirror the other (though I can't remember which).
• coffea.nanoevents.methods.vector, as we want to make the transition from Coffea to vector as seamless as possible.
• An email I've been keeping from @jrueb (sent to cms-coffea) about "Extending Nanoevents vector".

I'm mostly reiterating the goals of Vector as @henryiii has described them, but it has 8 principles:

1. 2D, 3D, and Lorentz, are distinct (non-inheriting) classes.
2. Arrays of vectors (arrays with behavioral methods) are distinct from individual vectors (simple Python objects), but they have all the same methods and behave the same ways.
3. Coordinate systems determine the member data that each class has: Cartesian, cylindrical, spherical, etc. These classes should be carefully arranged so that we can mix Cartesian vs polar azimuthal with η vs θ vs cos(θ) vs cot(θ) longitudinal with E vs m temporal.
4. Different storage/computation backends for arrays of vectors: NumPy, Awkward, and possibly CuPy, TensorFlow, Torch, and JAX.
5. Separate the mathematical functions from the backends, since the same Python expressions mostly duck-type everywhere. (The first argument of all of these functions should be lib, so that numpy can be replaced with cupy or jax.numpy, etc.)
6. Numba extensions for NumPy and Awkward, which also implies a lowered type for individual vectors.
7. Base arrays/objects have purely geometric names for their arguments, such as perp instead of pt, but overloads mix in synonyms.
8. Another mix-in keeps track of affine (position) vs vector (direction), mostly for strong-typing these concepts. For instance, affine - affine → vector, but affine + affine is an error, while all operations are available to vectors. (This is like ROOT's Math::PositionVector3D versus Math::DisplacementVector3D.)

The order of development will be roughly as follows:

1. Decide on a complete set of coordinate systems and name them. This will set a naming convention and scope for the mathematical functions.
2. Implement the mathematical functions, driven by tests against ROOT's Math::LorentzVector (through PyROOT). The functions can be tested before any classes exist by calling them directly in the test suite (unit, rather than integration, tests).
3. Develop the class hierarchy for NumPy only. This involves the inheritance structure of the momentum-name mixins and the affine-vs-vector mixins.
4. Extend that to CuPy as an easy example of a different library with a different lib to pass into all the math functions.
5. Develop the Awkward behaviors, which are not strictly methods on a class (some are through ak.behavior).
6. Lower the NumPy backend to Numba, which involves developing a lowered "individual vector" type.
7. Lower the Awkward backend to Numba, too, which also involves ak.behavior.
8. Maybe do JAX, too, because I've had some experience with that recently and differentiable Lorentz methods would be cool.
9. Type-annotate everything in one pass, just as I've found it useful to write all the docstrings in one pass.
10. Write all the docstrings in one pass, too.
11. Reinstate the protocols to ensure that no methods are missing or incorrectly typed.
12. Generate reference documentation from the docstrings (i.e. write .rst shells).

By this point, we'd want to release some versions and get some feedback from users. develop should be merged into main and releases/release candidates should be deployed. Hopefully, I can get to step 12 by the end of the month, but we'll see.

Last minute thoughts that I'll be thinking about tonight: do we need Transformations to be objects? A Transformation with both a translation and rotation would act on affines differently from vectors, and they would also be parameterized by coordinate system. (The vectors in ROOT's Math library have this.) Then would there be an array of Transformations to go along with an individual Transformation, like arrays? I'll have to think about whether that makes things easier or harder.

[nsmith-]

Re: transforms as objects, Lindsey floated boost tensors in coffea once so maybe worth having.

[jpivarski]

I read that and it helped firm my opinion.

My compromise between generality and the amount of work to do is this: Transformations should be first-class objects like Vectors, but Transformations don't have to exist for all coordinate systems (which would create an n² problem or worse with all the pairs of "my Vector is cylindrical, but my Transformation is spherical"). Transformations, which mix coordinates, should all be stored as Cartesian for simplicity. Vectors can store data in different coordinate systems, but they get converted to Cartesian when being transformed, just as some operations, like +, should collapse to Cartesian.

[nsmith-]

I think that's fair that there will be a set of operations (transforms, addition, perhaps others) that necessarily convert to cartesian storage. One nice feature might be a way for users to convert the storage back to spherical or otherwise, if it is known to be more efficient for subsequent processing.

[nsmith-]

Base arrays/objects have purely geometric names for their arguments, such as perp instead of pt, but overloads mix in synonyms.

If we are to make a fully separate set of positional and momentum property/method names, I think it would be nice to be able to optionally expose only one or the other.

[jpivarski]

It's hard to remove properties from an object, and doing so would involve some Python magic that probably won't play well with typing. It would be possible to get this by creating a momentum class that isn't a subclass of the geometric class, passing each member through, but then a lot of infrastructure would have to be duplicated for that because a lot of this works by inheritance (mix-ins).

I wonder, though, how important that would be. If you had a momentum vector or an array of momentum vectors with a pt property, is it a problem that it also has a perp property? After all, the geometric concepts are still valid when a vector is interpreted as momentum, but the reverse is not true. If you're using a vector for secondary vertex displacement, pt is potentially misleading, but perp is right, whereas if you're using a vector for momentum, pt and perp are both right.

[nsmith-]

One argument is the proliferation of attributes, having x, y, z, px, py, pz can get a bit crowded, especially if other record properties are comingled. But that is just aesthetics. Some objects, such as secondary vertices, have an associated position and momentum vector. There one might want to expose one or the other separately (though perhaps the answer there is to use a subrecord for each)

[jpivarski]

A secondary vertex (position in the detector) and the momentum sum of the particles produced at that vertex (momentum direction) really, really should be subrecords. Not just for this reason.

And anyway, ROOT Lorentz vectors have every synonym under the sun for every vector. By not including momentum-related words on non-momentum vectors and including pure geometric words on every vector, that is some clean-up. Arguably, we shouldn't clean up any more than that, because a momentum vector "is a" direction in a space, just not position space.

[henryiii]

I'm also perfectly fine with a minimal, basis set of properties based on pure geometric words; adding an interpretation doesn't need to invalidate the basis (it can't in Python, of course, but I don't think it even really ideally needs to).

[henryiii]

Let's avoid list 2 point 8 for now: as I've pointed out many times, I am strongly against using 3D vectors for positions; they are not points, they are directions + magnitude. The first iteration should not give any support to affine vectors at all. Functions like point + point are actually well defined - but not if you use a 3D vector for a position; because 3D position can't be mathematically represented with three components; only direction can! There will be two eventual possibilities; either someone can extend this for affine vectors in a standalone package (and they can take all the bugs associated with trying to do this with three coordinates), or we can eventually add a real 4D Cartesian vector and it's representation as a 3D point in space. There are well defined methods from computer graphics / ray tracing for working with points in this way.

[jpivarski]

Thanks for the reminder! I've been looking at the Scikit-HEP geometry packages—they're denoted with an S in a circle on the right side of this page (checking all the properties and methods of earlier vector packages, to make sure I haven't missed one; on the left is a set of collated names with synonyms).

[henryiii]

I would hope the Geometry package would become a standalone package dependent on Vector. Like GeometryVector or something similar.

[eduardo-rodrigues]

That's a possibility for sure. I just thought I would point the code out as it seems relevant to the discussions and design ongoing. You know from previous chats that I have no strong preference and would be totally happy to see line and like also here, though I see why you make the comment.

[eduardo-rodrigues]

BTW, @jpivarski, great notebook 👍. I still work with paper a lot myself - nothing like it!

[jpivarski]

I'm not planning on doing lines and stuff (math/geometry.py as opposed to math/vector.py). Actually, the scope is pretty well encapsulated by the above page.

[jpivarski]

I'm using the wiki to write up ontologies and naming conventions. This is mainly so that I can give names to a large number of compute functions and later will also be helpful in setting up the class hierarchy with appropriate mix-ins.

[jonas-eschle]

Great to see this and I'll gladly help with the implementation of TensorFlow and friends. I think they (jax, TF, PyTorch) share a commonality: they all have a numpy API, but don't allow in-place assignments (AFAIK), which we may want for e.g. pure Numpy.

[jpivarski]

Awkward Array doesn't allow in-place assignments, either, but when you're working with NumPy, you expect that. Therefore, the general classes won't assume but won't prevent mutability—that will be implemented independently on the backends that support it. That's how I was thinking about supporting it.

We should also keep an eye on the Consortium for Python Data API Standards, which is standardizing NumPy-like APIs (with the same view toward optional mutability). What they have in mind isn't general enough to include Awkward Array, but it would include all the backends, and maybe we can get a lot of them at once through some protocol this group develops.

[jpivarski]

A follow-up: the bulk implementation of Vector is pretty much done. I've written an overview here, which shows how the whole project fits together. Tomorrow, I'm writing docstrings.

At this point, we should start encouraging adventurous users to use it. Doubtless, there will be rough edges that need to be fixed—I found a few of them while writing the overview. I'll be pivoting to work on Uproot in April, and one of the things I ought to do is make Uproot present TLorentzVector, ROOT::Math::LorentzVector, and similar as Vector's object and array types.

Oh, we'd also need to transition the develop branch to main and publish a release. I wonder if @henryiii wants to look it over before taking that step.

Take a look!

[henryiii]

Yes, maybe we can aim for a release on Wednesday or so? I'd like to go over it, touch up a few things (like making sure the docs are showing up on RtD, currently untested, I think, due to the branch changes, making sure py.typed is present in the output wheel, etc).

[eduardo-rodrigues]

I'm sure to be having a look. From a quick glance it looks beautiful!

[jpivarski]

making sure the docs are showing up on RtD

This is missing: the .rst stubs will have to be recreated. I deleted the original ones because the module names are different now, but the new names (modules, classes, methods) are stable—there won't be any new ones before Wednesday.

Given the nature of what these functions do, I'll want to include some math formulas in the docstrings. I know that reST can do math, but it's horrible to read in text form. (I really don't like reST...) I'm going to see if there's something that will convert simple TeX between $ or $$ or something into the appropriate output for RtD, maybe a Sphinx extension.

[henryiii]

https://www.sympy.org/sphinx-math-dollar/