Generating callbacks for a c library

[adam-urbanczyk]

I'd like to generate functions like f(double*, int, ...) and use them with another library (e.g. nlopt). Is this currently possible with pyccel?

[adam-urbanczyk]

FWIW here is a more detailed example https://numba.readthedocs.io/en/stable/user/cfunc.html#dealing-with-pointers-and-array-memory

[EmilyBourne]

This example talks about scipy. It seems that scipy provides 3 options for exposing C functions to scipy internals (https://docs.scipy.org/doc/scipy/reference/generated/scipy.LowLevelCallable.html#scipy.LowLevelCallable):

• PyCapsule, whose name contains the C function signature
• ctypes function pointer
• cffi function pointer

Pyccel doesn't currently provide any of these options but of the 3 I think the PyCapsule would be the easiest to implement. Both ctypes and cffi seem to require Python code. There is either nothing available or nothing documented about the C-Python API side of things which means we would have to write C code which calls Python code in the setup.

I am not sure what other libraries use such a mechanism. It would be useful to compare several to see if there is a consensus about the standard way of doing this

[EmilyBourne]

Sorry, this discussion seems to have been missed. The functions that are generated can be used elsewhere as they are just basic C/Fortran functions, however I'm not sure if the input types would match what you would expect. E.g. for f(double*, int, ...) I imagine you have an array in mind, perhaps def f(a : 'float[:]'):. Currently (the type will change this year), a C translation of this function will have the prototype void f(ndarray_t a); which is probably not very helpful. Perhaps the simplest is to use the Fortran translation. The prototype is subroutine f(a) but the wrapper file will generate a function with the prototype void f(double*, int, int) (the additional int is for the stride, see #1608)

[EmilyBourne]

For nlopt called from Python, it seems you simply need to provide a Python function so I don't know if you need anything extra in this case.

[EmilyBourne]

@adam-urbanczyk I want to expand on this as I am unsure whether you have missed this or if I have missed something that would make ctypes necessary.

ctypes is usually used to expose a function in a low-level language like C to Python. The alternative ways of doing this are usually to use a C-Python API wrapper or cpython. (see discussion here: https://stackoverflow.com/questions/1942298/wrapping-a-c-library-in-python-c-cython-or-ctypes).
Pyccel already generates a C-Python API wrapper. As documented in the quickstart :

By default Pyccel also compiles the C code into a Python C extension module named mod..so, which is placed in the same directory as mod.py. To achieve this Pyccel generates the additional files mod_wrapper.c (which interacts directly with the CPython API) and setup_mod.py (which defines the build procedure for the extension module), as well as a build directory.

If the command import mod is now given to the Python interpreter, this will import the Python C extension module mod..so instead of the pure Python module mod.py.

This means that if you use nlopt by importing a Python function from another file and you run the Pyccel command line tool on this other file then subsequently nlopt will be using the accelerated/generated C function.

I can't find any mention of ctypes in the docs of nlopt so it doesn't seem that nlopt provides any way to bypass the Python interface. That being the case the wrapped function provided by Pyccel may well be faster than the ctypes function as the type checks are hard-coded instead of being determined on the fly.

[EmilyBourne]

We have not implemented proper callback-functions as documented here:
https://docs.python.org/3/library/ctypes.html#callback-functions

This is something that I would quite like to do as it would allow us to expose functions which take functions as arguments (currently they are translated but not wrapped. I hadn't realised that there was a standard way to do this.

[adam-urbanczyk]

Thanks, I opened #1900 to track this.

[EmilyBourne]

The more I look into this the less I am sure that ctypes is the right way to do this. In order to expose functions which take functions as arguments I wanted to start wrapping functions as classes which have a __call__ method. By doing this they can be called as before but can also be identified and can have a property that somehow stores the C function. For example, with ctypes it would be nice to have a low_level_callable property which is a CFUNCTYPE. However I can't find any documentation on how to build a ctypes object from the Python-C API. This is a little concerning as it may mean that either these functions aren't exposed in the API or they are not stable. That said the scipy code seems to use these without any trouble so perhaps we could use this for inspiration

[EmilyBourne]

There is a major problem with this suggestion though (as regards its use in external libraries like scipy) which is that the code which uses this low_level_callable property would then be different to the code that uses the original Python code. This would still be useful for us internally though for functions which take functions as arguments and would work as a good jumping off point for supporting external calls

[adam-urbanczyk]

I'm afraid that I do not follow. The context of my question was using pyccel for making shared libraries that could be called in other languages via a C API (specific example would be defining a cost function for nlopt, where nlopt is used from C and not from Python). I do not see how Python C API is relevant for such a use case.