Literal unroll of jitted functions containing ctypes functions fails without explicit signature (unclear error message).

[johnhelt]

Reporting a bug

• I have tried using the latest released version of Numba (most recent is
  visible in the release notes
  (https://numba.readthedocs.io/en/stable/release-notes-overview.html).
• I have included a self contained code sample to reproduce the problem.
  i.e. it's possible to run as 'python bug.py'.

I have some code, which has to call a list of functions in a loop. I know that Numba doens't handle heterogeneous lists, so the recommended approach is literal_unroll. The function I need to call, calls an external c-function. For this I am using ctypes to load the function from the library. The problem I experience occurs during compilation where numba throws an error: ctypes objects containing pointers cannot be pickled.

Here's an example of a code where this occurs:

import ctypes
from ctypes import cdll
from ctypes.util import find_library
from numba import literal_unroll, njit, cfunc, types
from numba.core.extending import intrinsic
import numpy as np

@intrinsic
def address_as_void_pointer(typingctx, src):
    """ returns a void pointer from a given memory address """
    from numba.core import types, cgutils
    sig = types.voidptr(src)

    def codegen(context, builder, sig, args):
        ptr = builder.inttoptr(args[0], cgutils.voidptr_t)

        return ptr

    return sig, codegen


dll = cdll.LoadLibrary(find_library('C'))

printf_ptr = getattr(dll, 'printf')
printf_ptr.argtypes = [ctypes.c_void_p]
printf_ptr.restype = None


sig = types.void(types.int64)

@cfunc(sig)
def printf_cfunc1(string_ptr):
    print("this is 1 cfunc")
    printf_ptr(address_as_void_pointer(string_ptr))

@cfunc(sig)
def printf_cfunc2(string_ptr):
    print("this is 2 cfunc")
    printf_ptr(address_as_void_pointer(string_ptr))

@njit()
def printf_njit_1(string_addr):
    print("this is 1 njit")
    printf_ptr(address_as_void_pointer(string_addr))

@njit()
def printf_njit_2(string_addr):
    print("this is 2 njit")
    printf_ptr(address_as_void_pointer(string_addr))


s = "hello world!\n".encode('utf-8')
s_ptr = np.array([s]).ctypes.data_as(ctypes.c_void_p)
s_ = s_ptr.value

@njit()
def printf_litunroll(string_addr, printf_list_):
    for printf_ in literal_unroll(printf_list_):
        printf_(string_addr)


printf_litunroll(s_, (printf_cfunc1, printf_cfunc2))
printf_litunroll(s_, (printf_njit_1, printf_njit_2))

Calling this function prints out:

this is 1 cfunc
hello world!
this is 2 cfunc
hello world!
...
ValueError: Failed in nopython mode pipeline (step: handles literal_unroll)
Failed in literal_unroll_subpipeline mode pipeline (step: performs mixed container unroll)
ctypes objects containing pointers cannot be pickled

So it calls the @cfunc decorated function ok, but not the njit decorated one. This lead me to try out adding the signature for the njitted function as well, and lo' and behold, the code runs if I add @njit(sig) as decorator for the njitted functions:

this is 1 cfunc
hello world!
this is 2 cfunc
hello world!
this is 1 njit
hello world!
this is 2 njit
hello world!

I cannot find anywhere in the documentation that says this is normal behavior. If anything the error code is wrong, because clearly we are able to call a ctypes function containing a pointer inside the literal_unroll.

[kc611]

Hi @johnhelt, Thank you for reporting the issue. I can confirm this behaviour locally and it does seem like inconsistent behaviour that needs addressing. Marking this as a bug for now.