Conversion between Java and Python collections

[ctrueden]

In #295 (comment) @KeyWeeUsr writes:

PyJNIus is not a complete wrapper for Java, therefore just use the appropriate Java methods for that such as java.util.ArrayList.set().

For the scyjava project, I implemented conversion routines between common Java and Python data structures, including: iterables, iterators, map/dict, lists, and sets. The code can be found here. The Python-to-Java conversions are done eagerly as copies. The Java-to-Python conversions are done as new Python-side classes that extend appropriate collections.abc classes and implement all the appropriate functions. This allows complex Java objects to be treated more like Python objects—e.g. you can easily iterate them using Python syntax.

My questions are:

1. Would pyjnius be interested in this functionality migrating upstream? At least as explicit conversion functions?

2. If so: what about passing a Python list or dict or set directly to a Java method call that expects a List or Map or Set respectively, without needing to explicitly convert to Java? To my naive eye, it looks like Convert Python array-like structures properly to Java Objects #386 by @KeyWeeUsr starts to go in this direction, although only between Python stuff and Java Object[]? Would the scyjava conversions just be new cases in jnius_conversion.pxi? Or somewhere else in the code? Or is it not that easy?

3. Finally—or perhaps in lieu of (1) and (2)—it would be super slick if the supported cases could be extended by third party code. Then I could register the supported scyjava conversions with that extension mechanism, and we would not need to migrate the code into pyjnius. To be clear though, I'd love to contribute this code to pyjnius—I just need a little guidance on where to put it, and I can make a PR.

Thanks very much for all your time and effort on the pyjnius project.

[tshirtman]

Hi, sorry for not answering before, i guess we have neglected this project quite a bit 😬
I'm pretty sure at least 1 is desirable, having an easy way to convert, even explicitly, would be a good first step. I don't have well formed thoughts on the desirability of 2, it seems nice, if that doesn't involve too much guessing (though pyjnius already does quite a bit of it for conversion of basic types), I feel there is space for experimentation, and if you want to do a PR for that, that can certainly be discussed, though it's more likely to be controversial than explicit conversion, but since i guess they don't need to be mutually exclusive, as 2 would certainly build upon 1.

3 seems interesting, but i'm not sure how a hook system would work in pyjnius, some kind of middleware register system? so everytime an object is exchanged between java and python, it goes through a list of registered function, that are free to return the same or a different object, before the resulting object is actually sent to the java side, or delivered to the python side? I'm not sure about the ease of use of such a system, if that's what you are thinking about, and i'm a bit worried about both the performances and the debugability of apps (ab)using this feature. But maybe i'm burning a strawman, so if you want to clarify, feel free to do so. 😄

[ctrueden]

@tshirtman Thanks for the feedback! I'll try to find some time to dig into the pyjnius code in more detail and explore the best way to include the conversion routines. As part of that, I'll think about what a possible hook system for extensibility could look like. One PR at a time. 😄

[yimelia]

@ctrueden
Thanks for your code!
It's very useful for me!

[ctrueden]

My community has switched to JPype (see also #551), which does more collections-related conversions. So the chance I'll follow up on this issue is now realistically zero. Therefore, I'm closing it to avoid clutter here.