I'm wondering if the use of an anonymous fun() should be denied with leader and consistent queries (i.e. only allowed for local queries). Otherwise, it might lead to rare issues that are difficult to debug.

@ansd, @dcorbacho, @the-mikedavis: Any opinion?

ra:local_query(ServerId, ...) means locally on the given ServerId, which may be a remote node. In that case, anonymous funs should still not be used.

Otherwise, it might lead to rare issues that are difficult to debug.

What are the rare issues that are difficult to debug when always using the {M,F,A} query_fun() format?

ra:local_query(ServerId, ...) means locally on the given ServerId, which may be a remote node.

Khepri uses local queries with the local node only:

LocalServerId = {StoreId, node()},
Ret = ra:local_query(LocalServerId, QueryFun, Timeout),

What are the rare issues that are difficult to debug when always using the {M,F,A} query_fun() format?

Sorry I was unclear.

I prepared a patch so that Khepri always use MFA. We should be fine on this front. However, APIs such as khepri:fold/5 may continue to receive fun() as argument. Thus I’m wondering if Khepri should accept fun() when doing a local fold only, and reject otherwise.

Yes, I think that's a good idea.

After putting more thoughts into this, I believe the problem is larger. In fact, there are two problems.

1. The validity of a function reference on a remote node
2. The possible difference of code between the local copy and the remote one

We could add one somewhat related issue: Ra machine versions compatibility.

The validity of a function reference on a remote node

This the one we attempt to fix here. Indeed, depending on the version of Erlang/OTP or the remote module hosting the function, the reference could be invalid and the Ra machine could crash trying to run it. This can be solved by always using exported functions and MFA as proposed in the current patch.

But this brings us to the second problem…

The possible difference of code between the local copy and the remote one

If Khepri or the application using it are updated on one of the Erlang nodes, the code of the query the caller thinks will be executed might be different or even incompatible with the code actually executed on the remote node. This could mean an unexpected result or a crash.

Ra machine versions compatibility

Speaking of upgrades, we have the same king of problem if Khepri’s Ra machine’s state changes. If the query code doesn’t know about the new or old state, again, it can give bad results or crash.

That’s why I’m pausing work on this patch and started to discuss solutions with @kjnilsson.