GIL usage

[jmirabel]

At the moment, all Pinocchio calls takes the GIL during Pinocchio calculations. This is sometimes problematic in multi-threaded Python applications.

For time-consuming operations such as collision detection, it would be convenient to have the ability to release the GIL when Pinocchio performs its calculations. Obviously, this needs the cooperation of the user. Objects used internally should not be used anywhere else.

[cmastalli]

This is indeed a problem we have in Crocoddyl as well. When we derive a Python implementation of an action model, Crocoddyl deactivates multithreading computations under the hook.

I tried to fix this issue a few years ago, but I miserably failed. Do you know how to do it? I am keen to fix this in Crocoddyl.

[jmirabel]

This is the opposite of the original question. I use C++ code in a Python app and I want the C++ code to run without locking the GIL.

You can C++ to call Python. For that to work, you have to make sure the thing that calls the Python function locks and unlocks the GIL at the right time. It should not be hard though it is tricky to debug.

[nim65s]

process based parallelism like ProcessPoolExecutor can bypass GIL issues.

I am not sure if eigenpy data would be serialized between threads, or only pointers to data. In the first case, we would need some shared memory + locks hacks to stay fast I guess.

[nim65s]

Then you don't need to share big amount of data between those threads, do you ?

Don't you think a message system like a Queue would fit your need ?

[jmirabel]

The interactions are a bit more complex than an input/output mechanism so it cannot be just a Queue. Process in Python can be very useful but are usually less comfortable to work with. Thread are easier to work with in general.

I'm not saying that I cannot use Python processes but I don't think it is meaningful in this discussion. The question is whether it makes sense for Pinocchio Python API to give the ability to perform GIL free calculations for time consuming algorithm.

[nim65s]

I think a GIL-free-public-API is a footgun. I am not strongly against it, but I think we must hunt XY-problems first.

If we are to add this, it should be clear for what purpose we do this, so we can start deprecating/removing it as soon as it is no longer relevant to our goals. This is why I think your current use-case is meaningful.

[jmirabel]

I don't mean to make all the API GIL free. What I mean is something like a pinocchio.gillfree submodule that contains some algorithms.

I can also do that on my own if you are reluctant to do it in the main repo.

[nim65s]

If we do this, I think it should be in the main repo, additional to what we currently have, and under an explicit namespace like your suggestion.

[nim65s]

CPython 3.13 will provide a --disable-gil compilation option. Would it work for you ?

ref. https://peps.python.org/pep-0703/

[jcarpent]

Thanks @jmirabel for raising this discussion.
Few years ago, I thought to implement that. I think it is a good feature to have, and not so hard to do.
I will handle this task on the week of May, 13th. I will add this feature to EigenPy to make it generically to all modules relying on Boost.Python.