Can we remove the dependence of the DB on jagged arrays?

[opotowsky]

Our dependence on jagged arrays means we have a max numpy version of 1.23.5.

What is the basis for needing jagged arrays? Can we restructure something to remove this?

We have a ticket open for this here.

@john-science @onufer @albeanth @mgjarrett @jakehader @sombrereau

[albeanth]

Do you know where we use jagged arrays?

[jakehader]

I'm not entirely sure the basis, but I have to assume it's to reduce DB size with zero values / sparse data in an array / matrix (e.g., Nones, NaNs)? @youngmit might come to our rescue here 😉. If you search "jagged" in the repository there are several instances in the DB implementation docs and comments @albeanth.

[mgjarrett]

There aren't actually any jagged arrays stored in the database. But Database3 sometimes creates a jagged numpy array before flattening it into a 1D array and storing that 1D array (along with offsets) in the database.

So we would probably just need to build our own JaggedArray class that would be a special type of list. You would create a JaggedArray instance, append a bunch of numpy arrays to the list, and then concatenate the whole thing into a flattened 1D numpy array with offsets and spit that back out to be dumped into the database.

[mgjarrett]

I'm looking more closely, and it appears that we also create jagged numpy arrays for convenience when reading the data back out of the HDF5.

armi/armi/bookkeeping/db/database3.py

Lines 1361 to 1363 in 827d223

	data = numpy.array(layout.location)[layoutIndicesForType][
	indexInData
	]

It would take some work to refactor all of this to not require a jagged numpy array, but it's probably not too complicated.

[john-science]

The problem I originally had in this ticket was identifying, conclusively, all the locations we use jagged arrays.

That would make this work significantly easier.

[opotowsky]

Ok this isn't as urgent as I thought. I tried to recreate the errors I thought I had on linux with numpy 1.23.5 and am unable to do so.

It's still a good idea to put into a longer term plan so we aren't locked into ever-aging versions of numpy, but instead of a this-year thing it's a next year thing ;-)