Categoricals should merge subject to particular rules

[agoose77]

Version of Awkward Array

main

Description and code to reproduce

• Categoricals should merge for strings, but error for non strings?
• We can add an extension for merging non-string categoricals.

[jpivarski]

This is going to be hard to understand in 12 months!

Here's what I remember of our conversation:

• merging X with X results in X (naturally)
• merging categorical[X] with X results in X (I think this is least surprise, and has the least-bad performance consequences: the second X could be a million distinct strings)
• merging categorical[X] with categorical[X] results in categorical[X], but the look-up categories have to be combined into a single look-up, without duplicates, which changes the indexes of at least one of the two.

The third case can be handled for the special case in which X = string or X = bytestring, but not in general, not unless the user passes some way of determining whether two instances of X are equal, or a way of ensuring that a set of instances of X has no duplicates (a new ak.behavior?).

For now, the third case should just go to NotImplementedError, so that it's a reminder that something better needs to be done there (and maybe point to this issue). In the planned future, merging-strings should eventually be implemented. The general case is in the unplanned future—we'll only do it if someone encounters this need. (So there may be a NotImplementedError case in the code forever, and that's fine.)

[agoose77]

Thanks @jpivarski — haven't got round to writing up my notes yet! Looks like you've saved me the challenge :)

[agoose77]

Just to add some additional context: we discussed making support for merging categoricals be a behavior-overload. It would probably make sense for this to only be defined for RecordArray, meaning that awkward should implement support for merging categoricals of the various primitive types that we support.