Prevent Lua VM re-entry through JIT trace.

[igormunkin]

TL;DR:

I'd like to upstream the patch (tarantool/luajit@4f4fd9eb) implemented almost 4 years ago to prevent Tarantool crashes caused by FFI machinery misuse (see tarantool/tarantool#4427 for more info), that is widely exploited in Tarantool sources. The Tarantool issue has been fixed, and no related crashes has occurred since the patch has been applied to the long-term branches within our LuaJIT fork. Hence, I decided to try landing this patch to the upstream considering the fact #1066 is finally aboard.

@MikePall, I'm OK if you don't want to apply the changeset; at least the misuse will be described here and someone exploiting any FFI holes (I'm completely against this) will know the symptoms and the pill.

---

Description

JIT recording semantics assumes FFI calls are leaf regarding the LuaJIT VM: if the execution exited Lua world through FFI machinery it is not re-entering Lua world again.

However, there is a way to break this assumption via FFI: one can re-enter LuaJIT VM via Lua C API, used within the particular C routine called via FFI. As a result the following host stack mix is created:

Lua-FFI -> C routine -> Lua-C API -> Lua VM

This sort of re-entry is not supported by LuaJIT tracing compiler. @mraleph named such kind of the call stack an "FFI sandwich" in the tarantool/tarantool#4427.

This changeset introduces the mechanism for Lua-C API callbacks similar to the one implemented for Lua-FFI: trace recording is aborted when the execution re-enters LuaJIT VM. If the re-enter is detected while running the particular mcode, the runtime finishes its execution with EXIT_FAILURE code and calls the panic routine prior to the exit.

The patch slightly differs from the one committed via tarantool/luajit@4f4fd9eb, but all the changes are cosmetic to make the original idea more clear.

---

How to reproduce the bug:

libsandwich.c

#include <lua.h>
#include <lauxlib.h>

struct sandwich {
   lua_State *L; /* Coroutine saved for a Lua call */
   int ref;      /* Anchor to the Lua function to be run */
   int trigger;  /* Trigger for switching to Lua call */
};

int increment(struct sandwich *state, int i)
{
   if (i < state->trigger)
   	return i + 1;

   /* Sandwich is triggered and Lua increment function is called */
   lua_pushnumber(state->L, state->ref);
   lua_gettable(state->L, LUA_REGISTRYINDEX);
   lua_pushnumber(state->L, i);
   lua_call(state->L, 1, 1);
   return lua_tonumber(state->L, -1);
}

#define STRUCT_SANDWICH_MT "struct sandwich"

static int init(lua_State *L)
{
   struct sandwich *state = lua_newuserdata(L, sizeof(struct sandwich));

   luaL_getmetatable(L, STRUCT_SANDWICH_MT);
   lua_setmetatable(L, -2);

   /* Lua increment function to be called when sandwich is triggered */
   if (luaL_dostring(L, "return function(i) return i + 1 end"))
   	luaL_error(L, "failed to translate Lua increment function");

   state->ref = luaL_ref(L, LUA_REGISTRYINDEX);
   state->L = L;
   state->trigger = lua_tonumber(L, 1);
   return 1;
}

static int fin(lua_State *L)
{
   struct sandwich *state = luaL_checkudata(L, 1, STRUCT_SANDWICH_MT);

   /* Release the anchored increment function */
   luaL_unref(L, LUA_REGISTRYINDEX, state->ref);
   return 0;
}

LUA_API int luaopen_libsandwich(lua_State *L)
{
   luaL_newmetatable(L, STRUCT_SANDWICH_MT);
   lua_pushcfunction(L, fin);
   lua_setfield(L, -2, "__gc");

   lua_pushcfunction(L, init);
   return 1;
}

sandwich.lua

local ffi = require('ffi')
local ffisandwich = ffi.load('libsandwich')
ffi.cdef('int increment(struct sandwich *state, int i)')

local cfg = {
 hotloop = arg[1] or 1,
 trigger = arg[2] or 1,
}

-- Save the current coroutine and set the value to trigger
-- <increment> to call the Lua routine instead of C implementation.
local sandwich = require('libsandwich')(cfg.trigger)

-- Depending on the trigger and hotloop values the following contexts
-- are possible:
-- * if trigger <= hotloop -> trace recording is aborted
-- * if trigger >  hotloop -> trace is recorded but execution
--   leads to panic
jit.opt.start("3", string.format("hotloop=%d", cfg.hotloop))

local res
for i = 0, cfg.trigger + cfg.hotloop do
 res = ffisandwich.increment(sandwich, i)
end
-- Check the resulting value if the panic didn't occur earlier.
print(res)

$ pwd
/LuaJIT
$ make -j
<snipped>
$ gcc -fPIC -shared -Isrc libsandwich.c -o libsandwich.so
$ LD_LIBRARY_PATH=. ./src/luajit test.lua 1 1 
3
$ LD_LIBRARY_PATH=. ./src/luajit test.lua 1 2
[1]    29035 segmentation fault  LD_LIBRARY_PATH=. ./src/luajit test.lua 1 2

As a result of the patch the last command ends the following way

$ LD_LIBRARY_PATH=. ./src/luajit test.lua 1 2
PANIC: unprotected error in call to Lua API (Lua VM re-entry is detected while executing the trace)

---

Co-authored-by: Vyacheslav Egorov vegorov@google.com
Co-authored-by: Sergey Ostanevich sergos@tarantool.org
Signed-off-by: Igor Munkin imun@cpan.org