Efficient way of filtering by random indexes in a large 3d-array?

[JBraig]

Came here after recommendation from this stackoverflow post.

Greetings,

My use case:

I'm working with FEM-files and looking for a way to store relevant data for the elements of the model (of which there can be a million or more in total). An example would be a data structure containing the principal axis system (as a 3x3 matrix) for each element, which I'd intuitively would try to implement as a nx3x3-array (or 3x3xn, which works better I'm told). Another one would be a 3x4xn-array, which contains the 3 coordinates (x,y,z) of the 4 corners of n tetrahedrons.

My problem:

I'm looking for a way to quickly filter these data structures by a list of indexes. In many cases, I only need a subset of those n elements. Other times, it's necessary to get only the values along a different axis (in the second example, I might only need the x-coordinates of the previously filtered elements.) So I'm looking for a way that let's me filter the elements efficiently according to my given subset, but also allows me to quickly access all elements along each axis (don't think that second part would even be an issue, but I figured it's better to mention it anyway to be sure).

Cheers

[apete]

There is no functionality to quickly filter data structures by an arbitrary list of indexes. Since these arrays are very big, don't you want to access elements using some pattern (that could be coded in a "slice" method)?

When/if you store the data in 3x3xn arrays, and then try to use things like:

1. array.matrices() – have to iterate through all matrices or until you found what you wanted
2. array.sliceSet(long[], int) - Possibly we can create additional "slicing" methods, but they need translate to a series of indices with uniform stride.
3. array.loop(Predicate<long[]>,IndexCallback) - not very efficient as it needs to test everything

How far does that get you?

[JBraig]

Unfortunately, there isn't a pattern I could use to access the elements, as they are in a more or less random order. The attributes I want to filter for change during the program, so there's no nice and regular order I could use, so the sliceSet-method is out.
I tried around with the matrices()-version a bit, and I'm a bit confused by its behaviour in general. I tried a simple iterable -> list conversion, but the resulting list seemingly doesn't contain each individual MatrixView, but only the last one. It seems like the offset value just defaults to the offset of the last iterable MatrixView? Is this behaviour to be expected?

[apete]

The MatrixView is just one instance that points do different parts of the underlying ArrayAnyD. If you want to save some of the matrices viewed you have to copy them before moving on to the next.

If MatrixView had a "goToMatrix(int)" method, would that solve your problem?

[JBraig]

I'm not sure whether it would solve the problem completely, but I think it would help. I admit that I might be a bit spoiled, given that the project I'm working on was originally written in Python and I now want to rewrite it in Java. In the original, I just used numpy-arrays for more or less everything, where these sorts of indexing operations were both very fast and very intuitive, as I could just do something like reducedArr = fullArr[inds,:,:]. But I don't know enough about the implementations to be able to say how they made it so fast.

[apete]

If you can build a snapshot version and try this:

https://github.com/optimatika/ojAlgo/compare/AnyD

[JBraig]

Sorry it took me so long to review the changes and reply, but that looks pretty good! I especially like the sliceSet-method, that should help me a lot. Many thanks!