PyJnius vs JPype

[Thrameos]

I am the lead author of JPype. As part of updating the documentation of JPype, I added PyJnius to the list of alternative codes to JPype. Unfortunately after two hours of playing around with PyJnius, I was unable to come up with anything that looked like an advantage of PyJnius over JPype. Every aspect that I looked at from Proxies, customizers, multidimensional array handling, javadoc integration, GC handling, buffers, scientific code integration (numpy, matplotlib, etc), caller sensitive methods, documentation, and even execution speed in most cases is currently covered in JPype more completely that PyJnius. However, being the author of JPype perhaps I am looking at aspects that I value rather than those of the PyJnius team. Can you better articulate the advantages of this project? What is the value proposition of this project, what is its target audience, and what is it targeted to do that alternatives don't already cover?

[tshirtman]

Thanks for reaching out!

I might misremember because it's been years i didn't look at jpype, but i though it was using a different approach, of talking to the JVM through a server (so IPC) rather than shared memory (but your readme hints at the opposite, and i don't see hints disproving that from a quick glance at the code, so i'm probably totally wrong), and this approach made it unworkable on android, for example (which was kind of the primary reason of developing pyjnius, although some people do use it on desktop platforms). On the other hand, i don't see much hint at android support in JPype codebase, unless that's what the native directory is about, as its jni.h seems to be taken from AOSP?

But honestly, we weren't aware of JPype at all when the project was started, it was just a step up from having to code jni code manually to interface to specific android classes for kivy support. I believe @tito did some digging to compare later, when we came to know about it, but i don't remember if he saw specific reasons not to try switching.

[tito]

Hi. I remember JPype name, but at the time of searching what could we use, i don't recall why it wasn't used at all honestly, 8 years ago :) The only goal at the start was to be able to communicate with Android API, but not using a middle RPC server as P4A project was doing at that time.

[Thrameos]

I did a reach out to all the Java bridge codes about 2 years back. Unfortunately, PyJnius code apparently got missed as it never came up in my search. I would have missed it on this round as well except I was searching for the page that did the nice tech press release last time and stumbled a blog discussing the two projects. Not sure how I missed another active project in the same area for two years, but it was clearly my fault.

Sounds like you got JPype confused with Py4J which is the other major bridge code. They do everything using sockets with both the benefits and downsides that entails. I similarly found that project did not meet my requirements.

I have not done any research on what is required to support android. Though if I have some technical specs than it should be possible. There is nothing that we are doing that is outside of native JNI and plain old calls to the C API of Python.

As far as the approach, JPype uses JNI exclusively to marry a JVM to Python using the "startJVM()" command. Though the next version (2.0) also will provide the ability to do the opposite in which Python can be started from within Java. It does this through a layers approach. There is a Python layer which holds all high level classes that serve as a front end, a CPython private module with base classes holding the entry points, a C++ backer layer that deals with all type conversions and matching as well as acting as the native module for Java library, and a Java library which all the utility tasks (holding lifetimes of objects, creating backing classes for slices and exceptions, and javadoc extractors/rendering).

8 years ago JPype was a bit of a mess. It was trying to support both Ruby and Python so the C++ layer was a tangle of wrappers to work with and the front end was all in Python so it was very slow. It was also bifurcated on the return types as numpy support could be compiled in resulting in a different objects being returned. It needed a lot of adapter classes like JException to act as proxies where the native Python and Java object differed. But all those issues have been resolved in the 3 years since I joined the project. The two primary goals (for me) in JPype is to provide syntax which is simple enough for physicists would are familiar with programming to be able to use Java and to have high levels of integration with scientific Python code. We do this by making those objects that are backed by Java to have "all the frills" CPython object wrappers. Rather than converting a Java primitive array, we make a Java primitive array a new native Python type which implements all the same entry points as numpy and all these are backed by memory buffer transfers. Thus we have fast conversions by calling list(jarray) or np.array(jarray).

To adapt it to Android the start up sequence would likely need to be reworked and the thunk code which it uses to load its internal library would need to be replaced with a more traditional JNI model. I have already removed the thunk code in the next version so the later has already been met. Only the former would be required.

Key differences in approach that I can see is that PyJnius converts arrays going to and from. This would appear to be very prohibitive to scientific coding where passing large arrays back a forth (often that never get converted) is the preferred JPype style. The decision to require conversion then forces options like pass by value and pass by reference, but that potentially leads to greater issues as if you have a multi argument call you are picking one policy for all arguments. It also would make dealing with multidimensional arrays difficult. (I think an adapter class used like obj.method(1, jnius.byref(list1), list2) would have provided better control if it can be achieved). In addition, there is a lot of problems that JPype has solved such as GC linkage, caller sensitive methods, and such. If nothing else please look over the JPype code and see if there are any good ideas that you can use.

[cmacdonald]

@Thrameos one recent thing we are looking to merge is the use of Python lambdas for Java Functional interfaces. See #515 ; I didnt see that in JPype?

[Thrameos]

JPype has support for lambdas from Functional interfaces starting at 1.0.0. It was part of the 30 pulls in 30 days March push to 1.0.

JPype has had a long incubation period. Originally started in 2004 running until 2007. It then had a big boost as the group of users resurrected it to port it to Python 3 around 2015. Then in 2017, it was picked up for use at a national laboratory which carried it from 0.6.3 to 0.7.2. During that period all efforts were focused on improving and hardening the core technology that provides the interface. But that was finally completed in March after the second core rewrite so we could finally make the push for 1.0.0. Since then we have been adding everything that was "missing" during my "30 pulls in 30 nights" wish list campaign. The backlog of everything which I just couldn't implement because it would be too much work was finally cleared (issues dropped from 50 to 20, solicit users to ask for what do they need, etc). So there may be a number of features that you didn't find in previous versions that are now available. I have been turning most feature requests over less than a week such that all I have left are the big 3 (reverse bridge, extend classes in Python, ability to start a second JVM).

The project will go back to slumbering as I am 2 months into a 6 month long effort to complete the reverse bridge code which will allow Java to call Python and generate stubs for Python libraries so that they can be used as Java native libraries. It uses ASM to build Java classes on the fly so that native support for Python can be achieved. Still not fully integrated like Jython but perhaps close enough that there won't be much difference.

[tshirtman]

Thanks a lot for the detailed explanations, and indeed, if anything, surely there are ideas we could use, and it would be worth studying the code, looking at the code earlier, what i saw seems quite clean both in code quality and structure of the project, so congrats on all the work. You are right about my mixup with Py4J, i guess when i looked at JPype, it must have been in the messy state you describe, and using it must have been way more complicated than PyJNIus was at this point.

Your points about passing by value/converting are very true, and triggered some recent discussion here as well, as @hx2A looked into how to improve performance, and made converting back to python types optional (as passing a throwaway list to java, getting it converted to a java list, modified by java or not, and converted back to python, just to garbage collect it, was certainly suboptimal, we can now at least avoid the second part, at the cost of using keyword arguments, which is safe since java doesn't support them, so there is no signature clash, but it's certainly a bit more noisy syntax-wise).

As for a possible difference between JPype and PyJNIus, we can implement java interfaces using python classes and pass them to java to be used as callbacks, on the other hand, we would indeed need to generate java bytecode, if we wanted to extend java classes from python, as it's a requirement to use some android api, that we can't cover right now, i'm not sure if i infer correctly from your comment, but maybe you don't have that ability to have java classes call your python code like this (using interfaces).

[Thrameos]

JPype can implement interfaces in Python. Just add decorators to ordinary Python classes.

from java.util.function import Consumer

@jpype.JImplements(Consumer)
class MyConsumer:
   @jpype.JOverride
   def apply(self, obj):
       pass

Use a string in @JImplements if the class is defined before the JVM is started, multiple interfaces can be implemented at once but they all methods are checked to see if they are implemented. The major difference being that JPype uses dispatch methods (all overloads go to the same method) rather the overloaded methods. This is because we like to be able to call the methods from both Java and Python. I can add individual overloads if that is a desired feature, but no one has requested it.

(Edit: The reason we did not use Python inheritance is that at the time this was added we were still supporting Python 2 which caused a lot of meta class issues, we will clean is up further once class extensions are in place. So annotation which evaluate once at declaration time were cleaner.)

We use the same system to implement customizers (dunder?) for classes

@jpype.JImplementationFor("java.util.ArrayList")
class ArrayListImpl:
    def __getitem__(self, i):
        return self.get(i)
    @jpype.JOverride
    def addAll(self, list):
        # Decide if we need to convert or can call directly.
        ...

We also use annotations to define the implicit converters such as "All Python Path objects convert to java.io.File"

 @jpype.JConversion("java.io.File", instanceof=pathlib.PurePath)
 def _JFileConvert(jcls, obj):
       Paths = jpype.JClass("java.nio.file.Paths")
       return Paths.get(str(obj))

Obviously using this sort of logic is a bit slower then special C methods, but it keeps the readablity and flexibility high. I have been pushing critical paths that have proven to be bottlenecks back to C as needed.

We also provide some syntax sugar to make things clean MyJavaClass@obj => cast to MyJavaClass (Java equivalent (MyJavaClass)obj) or cls=JInt[:] => create an array type (cls=int[].class) or a=JDouble[10][5] => create a multidim array (double[][] a = new double[10][5]).

I have been working on the prototype for extending classes from JPype. We have the same problem as some Swing classes and other APIs required extending classes. The solution I have cooked up thus far it to create an extended class with a HashMap for each of the overridden methods. If there is nothing in the hashmap for that entry point then passes it to super, otherwise it calls the proxy method handler. But I decide that this would be easiest to implement after the reverse bridge is complete so that Java could actually handle Python methods rather than going through the Java proxy method. So it still about 6 months off before I will have the prototype working. You can look at the epypj branch (my name for the reverse bridge) to see how calling from Java back to Python works as well as patterns for generating an invoker using ASM to make Java classes on the fly.

As for managing garbage collection, there are few pieces of JPype that do that work. For first is the JPypeReferenceQueue (native/java/org/jpype/ref/JPypeReferenceQueue) which binds the life of a Python object to a Java object. This is used for creating buffers and other things where Java needs to access a Python concept for a duration. The second is the to use global references so that Python can hold a Java object in scope. This requires the garbage collector link (native/common/jp_gc.cpp) which listens for either system to trigger a GC and pings it to the other when certain conditions (size of pools, relative growth). Last Proxies need to use weak references because they would otherwise form loops (as the proxy holds a reference to the Java half and the Java half points back to the Python implementation). Eventually I intend to use an agent to allow Python to traverse Java but that is down the road.

[hx2A]

I'm one of the people using pyjnius on the desktop and not on android. I didn't know about JPype when I started building my project but I did some investigation to see what the differences are.

One unique feature in pyjnius is the caller can decide whether or not to include the protected and private methods and fields. My preference is for public only, but I understand the argument that making the non-public fields and methods available is useful.

Performance is critical to my project. I did some tests with the below class:

package org.pkg;

public class MyClass {

  public MyClass() {
  }

  public int number = 42;

  public float add1(float x, float y) {
    return x + y;
  }

  public float add2(float x, float y) {
    return x + y;
  }

  public float add2(int x, int y) {
    return x + y;
  }
}

In JPype:

In [1]: import jpype
   ...: import jpype.imports
   ...: jpype.startJVM()
   ...: from org.pkg import MyClass
   ...: myInstance = MyClass()
   ...:

In [2]: %timeit myInstance.number
640 ns ± 2.65 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [3]: %timeit myInstance.add1(10.3, 20.5)
2.13 µs ± 24.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [4]: %timeit myInstance.add2(10.3, 20.5)
2.19 µs ± 9.41 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In pyjnius:

In [1]: import jnius

In [2]: MyClass = jnius.autoclass('org.pkg.MyClass')

In [3]: myInstance = MyClass()

In [4]: %timeit myInstance.number
161 ns ± 0.104 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [5]: %timeit myInstance.add1(10.3, 20.5)
1.04 µs ± 8.16 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [6]: %timeit myInstance.add2(10.3, 20.5)
2.71 µs ± 11.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Pyjnius is quite a bit faster, except for the overloaded method. We should compare notes on how to decide which method to call when the method is overloaded. Pyjnius has this scoring mechanism which seems to add a lot of overhead. JPype makes that decision much faster.

Finally, for benchmark purposes:

In [9]: def add(x, y):
   ...:     return x + y
   ...:

In [10]: %timeit add(10.3, 20.5)
82.9 ns ± 0.187 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Of course, differences of a few µs are trivial but that does add up if one is making thousands of little calls very quickly, which I need to do.

JPype's integration with numpy is quite nice and easy to use. I can see how researchers could use this to write scripts that pass large arrays to Java libraries without complicated syntax. I also need to pass large arrays and I do that with tobytes() and special Java code that can receive the bytes, but obviously that is not as neat or convenient.

[Thrameos]

JPype speed unfortunately is sort of an is-what-it-is prospect. JPype is very defensive trying to guard against bad things which means there is a lot of non-trivial overhead. That means for example whenever a Java call is made it makes sure the thread is connected so we don't segfault if called from an outside thread such as an IDE. My local users group are all scientists so entry points are all very much guarded against some pretty horrible cross wiring. If something ever segfaults on them (no matter how crazy it was), then I have failed. (Which would explain the 1500 tests including deliberately constructing bad objects.)

Second, the speed of individual actions likely varies a lot in very small differences in work being performed. The trivial example you gave was one of the worst cases. Depending on what type of field is being access may actually change the speed of access in some cases.

For your speed example, you are requesting an int field.

• In PyJnius it creates a descriptor for the object lookup, accesses the field, creating a new Python long and then returning it.
• In JPype, it creates a descriptor for the object lookup, accesses the field, creates a new Python long, then creates a wrapper type for a Java int, copies the Python long memory to the JInt (because Python lacks a way to create a derived integer class directly), then binds the slot with a Java value, and finally returns the resulting JInt.

So even something as trivial as doing speed benchmarking on accessing a field, is not really so trivial.
The difference being one returned a Python long and the other returned an actual Java integer (that follows Java conversion rules) which could be passed to another overloaded method and bind properly. The work to return a wrapper type is a lot more than simply returning a Python type, hence a huge difference in speed.

I tried to demonstrate this by testing a few different field types. Unfortunately, when I tested Object fields, jnius segfaulted on the code "harness.objectField = harness". I am not sure why that particular piece of cross wiring caused an issue, but it failed for me. I haven't been much interested in speed for JPype beyond removing the gross offenders which gave a factor of 3-5 speed up for calling and 300 times for certain array accesses. But perhaps I should review and see what areas can be improved. I doubt though that I can reduce it to the bare bones like PyJnius without removing safety or remove return contracts (which I can't do). At most there is 10-30% speed up still possible,

As for the feature of accessing private and protected fields, it is certainly possible to do. I prefer the users use reflection or other internal access methods rather than exposing the object directly. If I had to provide something like that I would likely create a field called _private which would contain fields that are not exposed publicly. JPype only provides one class wrapper for each type, so I don't have much in the way of fine grain controls. So you couldn't select to create a class that has private access and then create a second object of that same type and not end up with the privates exposed. I went down that road with string conversion, and it was a disaster with some libraries choosing one policy and others a different one resulting in incompatibilities.

I ran some tests using array list.

import jpype
import timeit
jpype.startJVM()
ArrayList = jpype.JClass("java.util.ArrayList")

def pack():
    ja = ArrayList()
    for i in range(1000):
        ja.add(i)

def iter(ja):
    u = 0
    for i in ja:
        u+=i

def access(ja):
    u = 0
    for i in range(len(ja)):
        u+=ja.get(i)

def access2(ja):
    u = 0
    for i in range(len(ja)):
        u+=ja[i]


ja = ArrayList()
for i in range(1000):
   ja.add(i)

print("Pack arraylist %e"%( timeit.timeit("pack()", globals=globals(), number=1000)/1e6))
print("Iterate arraylist %e"%(timeit.timeit("iter(ja)", globals=globals(), number=1000)/1e6))
# Get is a direct call
print("Access(get) arraylist %e"%(timeit.timeit("access(ja)", globals=globals(), number=1000)/1e6))
# [] is emulated
print("Access([]) arraylist %e"%(timeit.timeit("access2(ja)", globals=globals(), number=1000)/1e6))

JPype

Pack arraylist 2.768904e-06
Iterate arraylist 5.208071e-06
Access(get) arraylist 4.037985e-06
Access([]) arraylist 4.690264e-06

Jnius

Pack arraylist 3.322248e-06
Iterate arraylist 4.099314e-06
Access(get) arraylist 5.653444e-06
Access([]) arraylist 7.762727e-06

It is not a very consistent story other than to say they are probably trivially different except where they are providing very different functionality. If all you are doing is accessing methods, then they are likely pretty similar. Passing arrays which are preconverted JPype is 100 times faster, convert lists and tuples JPype is 2 times slower (we don't use vector access or have special bypasses for tuples currently). So the bottom line is depending on your coding style it may be much fast with one or the other. But then my users usually select JPype for ease of use and bullet proof construction rather than speed. (Ah who am I kidding! They are just as likely stumble in off the internet because JPype happened to be the first link they found on Google.)

[Thrameos]

As for how JPype does method binding that detail should be in the developer/user guide. The method binding starts by presorting the list of methods for the dispatch according to the rules given in the Java spec so that if something hides something else it appears first in the list (code can be found in native/java/org/jpype as we use a Java utility class to do sorting when the dispatch is created the first time). In addition each method is given preference list of which method hides another. Resolution starts by first checking each argument to see if there is a "Java slot" for the argument. Java slots point to existing objects that need no conversion so getting these out of the way prior to match means we can use direct rules rather than implicit ones. It then matches argument based on the types into 4 levels (exact, implicit, explicit, none). It short cuts on explicit and none to jump to the next method. If it ever gets an exact then it short cuts the whole process to jump to the call. If there is a match it would hide the methods which have less specific binding. If two implicit matches are found that were not hidden it proceeds to a TypeError. Once all the matches are exhausted the conversion routines are execute. It then frees the Python global lock and makes the call and reacquires the global lock. The return type is looked up and a new Python wrapper is created based on the type returned (it uses covariant returns so the type returned is the most derived rather than the type from the method). This is mostly linear with the number of overloads though there are some complexities for variadic arguments, but the pre-constructed tables mean it would try foo(long, long) before it would try foo(double, double) and a hit on (long, long) would prevent double, double from every getting matched because of Java method resolution rules. There are still a few speed ups that I can implement but that would require additional caching tables.

I inherited the ordering system with short cuts when I started on the project in 2017. I added the method hiding cache and java slots to push out most of our overhead.

[Thrameos]

I optimized the execution path for methods. The revised numbers for JPype are

Pack arraylist 2.226081e-06
Iterate arraylist 4.082152e-06
Access(get) arraylist 2.962606e-06
Access([]) arraylist 3.644642e-06

[hx2A]

My local users group are all scientists so entry points are all very much guarded against some pretty horrible cross wiring. If something ever segfaults on them (no matter how crazy it was), then I have failed.

Yes, segfaults are horrific and I got hundreds of them when I started using pyjnius. I haven't gotten any in a long time though because perhaps I worked out the safety issues and built it into my code. Now everything works reliably. I understand your use case though. If your users are scientists working with the Java objects directly to do data analysis with various Java libraries, a segfault would cause them to lose all their work. JPype does seem better designed for doing scientific work where end users are working directly with the Java objects through Python. The main use case for pyjnius is different though, which is communicating with Android. In that case, the safety issues are the developer's problem, so perhaps making different choices about safety vs speed are appropriate.

[Thrameos]

I admit I am not a big fan of "it is safe to so long as you step on these squares in this order". When I started working on JPype it took me close to a year to bullet proof all of the entry points enough that I could pass the code around to my local group. And I have added two additional years of API armor since. Other than a few rare people whose JVM fails to load (which is very hard to resolve) there is few remaining issues as JPype has been brought up to production code standards.

As for as a trade off with speed and safety, speed is great but if you are getting speed by skimping on safe operations is it usually a poor trade for most users. Whether you are turning around prototyping code or writing a production system, having to stop work and try to work around a segfault is a distraction users shouldn't be facing.

If someone is willing to give me some examples of how to test JPype on a Android emulator, I can see about making the required modifications.

[tshirtman]

To use on android, we package pyjnius as art of a python distribution built by python-for-android, (often using buildozer as a easier interface to it, but it's the same), and then building a python application that ships this distribution, then your python code can import pyjnius or any other python library that was built in the python distribution when user runs the app.

The first step, thus, is to have jpype compiled in a distribution, which is done by p4a (https://github.com/kivy/python-for-android/) when you tell it it's part of your requirements, usually (but not always) a "recipe" is needed to explain to p4a how to build libraries that are not pure python, the one for pyjnius can be found here https://github.com/kivy/python-for-android/blob/develop/pythonforandroid/recipes/pyjnius/__init__.py as an example . If you use buildozer, you can use the p4a.local_recipes setting in buildozer.spec to declare a directory in which recipes for requirements can be found, so you don't need to fork python-for-android to have your recipe used.

I'd advise using buildozer, as it automate more things https://buildozer.readthedocs.io/en/latest/installation.html#targeting-android and you can still setup your local recipes to try things. The first build will take time, as it needs to build python and a number of dependencies for arm, and it'll need to download android ndk and sdk for it. You can probably use the default kivy bootstrap for the app, and create a "hello world" like app, that would import jpype and just display the result of some code in a label, or even in the logcat using print, i don't remember how well kivy runs in the android emulator, i never used that, but i think some users did, and with acceleration setup it should work, afaik, otherwise you could use the sdl2 wrapper, or the webview one, and use a flask or bottle server to display things, i'd first try with the kivy one as it's the most tested by far.

You'll need a linux or osx machine (VM is fine, and WSL on windows 10 is fine), to build for android.

[inclement]

If you start working on a jpype recipe for python-for-android, it would be welcome to open an in-progress PR about it for any discussion that may come up. It would be great if it would work, especially if it can indeed resolve some long-standing pyjnius limitations. As discussed earlier in the thread, pyjnius essentially covers our core requirements for kivy's use, but it doesn't have enough development oomph to significantly go beyond this.

[Thrameos]

@inclement I set up a PR for the android port at jpype-project/jpype#799. Unfortunately, I am not really sure where to go from here. It seems to be trying to run gradle which is really not the right build path.

The actions the need to be done are as follows:

• Include all of the jpype/*.py files in the build (or pre compiled versions of them).
• Run Apache ant on native/build.xml and place the resulting jar file someplace where it can be accessed.
• Include the jar file (or equivalent) in the build.
• Compile the C++ code from native/common and native/python into a module named _jpype to be included in the build.
• Include a main.py file which just launches an interactive shell so that we can test this manually for now.
• In the future I need to include "ASM" or something that works like it for android so that I can load dynamically created classes.
• Patch the C++ code so it uses a custom bootstrap to load the JVM and companion jar file and hook up all the native methods.
• Patch the jvmfinder with something that functions on android and "startJVM" is called automatically rather than at the start of main.
• Patch org.jpype so that jar navigation system (which is how imports work) can function on android.

I looked over some of the docs but nothing really stood out as to how to accomplish this. My project layout is somewhat different than normal as we don't put everything under the main module (as we are actually constructing 3 modules that make up the system. jpype, _jpype, and org.jpype.) I likely need a custom recipe to perform all these actions as well as disable undesirable patterns such as running gradle (unless it is doing something useful that I can't tell.)

[inclement]

It seems to be trying to run gradle which is really not the right build path.

Gradle is the build tool used as the final step of packaging an APK, it probably isn't related to your inclusion of jpype.

Include all of the jpype/*.py files in the build (or pre compiled versions of them).

In general, if jpype essentially performs as a normal Python module then your first recipe attempt there probably does most of the heavy lifting - the CppCompiledComponentsPythonRecipe does something like running python setup.py build_ext and python setup.py install using the NDK environment. This should install the jpype python package within the python environment being built for inclusion inside the app.

Include the jar file (or equivalent) in the build.

This is probably an extra step the recipe will need to do, it will come down to copying your jar file (or whatever you need) into some appropriate place within the Android project python-for-android is building.

Compile the C++ code from native/common and native/python into a module named _jpype to be included in the build.

If this is handled by the setup.py, this should already be working, but might need some tweaking. If it isn't, you can include your compile commands in the recipe (and have them build for the android environment by setting appropriate env vars, as you'll see done using self.get_env in other recipes).

Patch the C++ code so it uses a custom bootstrap to load the JVM and companion jar file and hook up all the native methods.

I hope this part should be fairly straightforward, just depending on using the right android JNI interface function. We do this in a slightly hacky way by patching pyjnius rather than doing some appropriate conditional compilation, as different helper libraries provide different wrappres, as demonstrated by e.g. this patch. This complexity wouldn't need to affect you, you can just call the right android api function.

Patch the jvmfinder with something that functions on android and "startJVM" is called automatically rather than at the start of main.

I'm not familiar enough with the JVM to really know, but I think Android wants you to always access an existing jvm and you can't start a new instance. Will that matter (or is it just wrong?)?

[inclement]

Run Apache ant on native/build.xml and place the resulting jar file someplace where it can be accessed.

I'm also not sure about this one due to unfamiliarity, is this just an internal jpype build step? I'm not sure how java versions interact, but I guess using the system-native ant should be fine here.

[Thrameos]

It gave a warning the buildozer does not respect setup.py so it skipped directly from the top to gradle step. Hence the need to add those middle steps manually. Our setup.py does a bunch of custom compiling steps such as creating the hooks file to merge the jar file with dll which likely are not applicable here so it was fine that it got skipped, but it also failed to locate the cpp and java files which are defined in setup.py. Part of the issue is that my setup.py was so large that I have had to split into the module setupext.

I guess first I need to figure out how I can run a clean command so that I can get the process to run once and show the log. (I ran it piece meal the first time as I was playing. so I don't have a clean log.) Also we should like move this conversation to the PR so that we can stay more on topic for the thread.

[hx2A]

FWIW I was able to port the core of my non-android project to JPype. There were two minor difficulties or differences:

1. The classpath kwarg in jpype.startJVM() seems to be ignored no matter what I do. The addClassPath() function did work though. Developers who care about the classpath order may need to make some adjustments compared to how they used jnius_config.add_classpath().

2. Implementing Java Interfaces works great but is a little bit different. In my code, I had a function that was returning a list of Python strings. In retrospect I probably should have converted each string to a Java string, but I didn't think of that and made the interface function definition return a list of Java objects to get this to work. That works fine in pyjnius, which effectively makes python strings a subclass of Java objects. That doesn't work in JPype, but I easily fixed that by converting the strings with JPype's JString class.

Writing @JOverride for the implemented methods is way simpler than code like @java_method('(Ljava/lang/String;[Ljava/lang/Object;)V').

Once I dealt with those two issues, everything basically worked just fine. I do have some code passing byte arrays that will have to change but JPype has good support for that. Also, JPype's implicit type conversions are super convenient and could replace a lot of the decorators I had to sprinkle everywhere.

@Thrameos I do respect your determination here. Best of luck getting JPype working on Android.

[Thrameos]

Thanks for the comments.

Not sure what could be wrong in the classpath though if I had to guess it would be where you were starting Python from. Sometimes people start the JVM from a module and because the rules for Java paths and Python paths with respect to the starting directory are different it can cause jars to get missed. (There is a section in the usersguide regarding this issue).

I use the classpath feature on a daily basis and it is part of our test patterns. So if classpaths didn't get respected it would cause some pretty major fails. That said you wouldn't be the first to have difficulties with finding the magic set of locations and absolute paths required to get a complex pattern to work.

Here is an example where I am starting up my test system from a Py directory relative to by development area.

import jpype
import os

devel = os.path.dirname(__file__)
devel = os.path.join(devel, '..', '..')
devel = os.path.abspath(devel)  # Notice that I converted the path to absolute so that it doesn't matter where the
# PWD of Java will be when this script is called.   Otherwise, if I import this from a different location it will use the
# original PWD and Java will find nothing in the classpath.

classpath = [
    '%s/gov.llnl.math/dist/*' % devel,
    '%s/gov.llnl.rdak/dist/*' % devel,
    '%s/gov.llnl.rnak/dist/*' % devel,
    '%s/gov.llnl.rtk/dist/*' % devel,
    '%s/gov.llnl.rtk.gadras/dist/*' % devel,
    '%s/gov.llnl.rtk.response/dist/*' % devel,
    '%s/gov.llnl.utility/dist/*' % devel,
    ]

jpype.startJVM(classpath=classpath, convertStrings=False)

Relative paths and absolute ones do work, but this will depend a lot on where you start from. The addClassPath has special magic to make sure that all paths are with respect to the caller location. I guess the same logic is needed in the classpath keyword arguments.

I can see the issue with returning a list of strings. It will depend on the type of the return as to the behavior that gets triggered. If the method was declared as returning String[] I would expect to force each Python string into a Java on when returning. If the method was declared as returning List<String> there would be an issue as Java generics strip it off so it would be List<Object>. Depending on how JPype views the list, it may or may not try to convert but that should be defined behavior so I will check it. The safe solution is a list comprehension that should be able to convert all of the items for the return.

[hx2A]

Not sure what could be wrong in the classpath though if I had to guess it would be where you were starting Python from. Sometimes people start the JVM from a module

That's what I was doing!

I use the classpath feature on a daily basis and it is part of our test patterns. So if classpaths didn't get respected it would cause some pretty major fails. That said you wouldn't be the first to have difficulties with finding the magic set of locations and absolute paths required to get a complex pattern to work.

Yes, I figured this is something you would notice right away and that I was not the first person to have this issue. Thank you for the code sample, that is helpful. I made some improvements to my code as a result.

The safe solution is a list comprehension that should be able to convert all of the items for the return.

That's exactly what I did and it works correctly now. Thanks!

[AbdealiLoKo]

A few years ago I had done a comparison between py4j and jnius and found jnius to be much faster.

When ever I saw jpype mentioned in places, I always assumed it was referring to - http://jpype.sourceforge.net/ and stayed away from it cause it seemed unmaintained (Didn't see the new project weirdly)

On reading this thread, I tried my benchmarks again (my primary testcase for speed is to read and score PMML files) - and found jpype was pretty performant.
Some results:
https://gist.github.com/AbdealiJK/1dd5b7677435ba22f9ab3e26016bb3e7

# jpype
# createjvm: 0.550s
# loadmodel: tot=1.466451 max=1.064521s avg=0.014665s
# fields   : tot=0.019881 max=0.009795s avg=0.000199s
# score    : tot=0.033356 max=0.023338s avg=0.000334s

# jnius
# createjvm: 0.249s
# loadmodel: tot=1.773011 max=1.385274s avg=0.017730s
# fields   : tot=0.039058 max=0.012234s avg=0.000391s
# score    : tot=0.067590 max=0.031904s avg=0.000676s

# py4j
# createjvm: 0.222s
# loadmodel: tot=0.616913 max=0.027464s avg=0.006169s
# fields   : tot=0.699152 max=0.026426s avg=0.006992s
# score    : tot=0.389583 max=0.017620s avg=0.003896s

[Thrameos]

To be fair, JPype had its front end API written in Python (as opposed to CPython) up until March 2020. So there are plenty of older benchmarks to show that it was rather slow for what it provided. There were simply so many backend issues that had to be resolved with memory management, removing whole multilayer wrappers to support Ruby, multithreading, and the like, that speed was the last thing to be worked on.

At this point it should be pretty speedy. But if you do find something that needs additional work relative to other packages just drop a note in issues. Because of return contracts there are some limitations, but the primary paths for calling, array transfers, and memory buffering are pretty worked over at this point.

[hx2A]

I'm also getting good results with JPype. In particular, I'm getting good value out of the integration with numpy arrays. I've been able to add new features I couldn't do before that have enabled me to improve performance in meaningful ways.

[Thrameos]

If anyone can be so kind as to try to update the kivy-remote-shell tool with to get it up to build with the current Python3 and buildozer and improve the instructions to be more step by step, it would help with the JPype porting effort greatly. I have completed all the requisite steps up to the boot and testing phase. But without an environment to complete the bootstrapping process, it will be tough to make progress. I can try again to update the remote shell tool this weekend (and maybe succeed) or an interest party with better knowledge of kivy can complete this prerequisite task and then I can spent the weekend completing the technical work that I am the most qualified to complete. Though I do freely offer my time to help others, it is a finite resources and any work that I do on android porting efforts is a delay on the Python from Java bridge which a number of other people are also interested in.

I hope the android porting effort can avoid going the way of the PyPy porting effort where I spent several weeks reworking the core code to be able to handle the differences but then hit a technical issue where a trivial difference in the object system produced an error, and I was unable to find anyone who could assist me to trace down how to debug an error report generated in generated code. Though I don't cry about spilled milk and all of that effort was folded in to improve the JPype code to other meaningful ways, at the end of the day those users who wanted to use JPype were left high and dry. If this effort fails not all is lost as I will cycle back to it, but once something is at the tail end of the queue I am hard pressed to get back to it for 6 months unless someone is able to put in time to assist me.

[Thrameos]

Progress update for interested parties.

I successfully got JPype to boot from within pythonforandroid and was able to test the basic functionality. Though some of the advanced features may not be possible due to differences in the JVM I believe that the vast majority of JPype will be available for use on the Android platform. The port took a while as it did require some upgrades to the buildozer and pythonforandroid projects (thus lots of reading through the source and asking for help). Many thanks to the developers here for being responsive so that I could complete the hardest part of the process in one weekend. It wouldn't have been possible without your input. I have put the related changes in as PR but looking at the backlog of PR, it may be a while before it comes up for consideration. Now that I have the key technical specifications that I need, I should be able to integrate it and have a working release code somewhere around JPype ,1.2 nominally on the calendar for late fall. I can push it forward if there is great user interest but it is competing with the Python from Java which is an important feature for other projects.

If someone would like to help accelerate the effort, the next hard step will be figuring out how to construct a docker image with everything in place with a partially build system so that we can execute a build, load an emulator, and run the test bench of the android on azure pipelines (or some other CI system). Once we have a working CI that can detect what works and what doesn't we will be much closer to being able to deploy as stable software. Not sure if this should be housed in the JPype project or if we should have a separate android test project.

[mattip]

hope the android porting effort can avoid going the way of the PyPy porting effort where I ... was unable to find anyone who could assist me to trace down how to debug an error report generated in generated code ...

@Thrameos a bit off topic, but please ping me in the future for PyPy related issues.

[Julian-O]

This is an interesting discussion, and I don't want to interrupt it - it can continue here.

However, it doesn't belong as an open issue; there is no action that needs to be taken. So I am closing it to clear it off the To Do list..