xpcall error handler result gets coerced incorrectly

[technomancy]

When you call error inside an xpcall and your error value is not a table, no matter what you return from your error handler, the resulting error will always be turned into a string.

local function err_handler() return {} end

local ok, err = xpcall(function() error({}) end, err_handler)
print("table err", type(err))

local ok, err = xpcall(function() error("oh no") end, err_handler)
print("string err", type(err))

You can see the difference between luaj and PUC lua:

$ luaj scratch.lua
table err	table
string err	string

$ lua scratch.lua
table err	table
string err	table

This bug prevents the luaunit test framework from working with luaj: https://luaunit.readthedocs.io/en/luaunit_v3_2_1/

[amsitlab]

but value is an table

local function err_handler() return {} end

local ok, err = xpcall(function() error({}) end, err_handler)
print("table err", err, type(err))

local ok, err = xpcall(function() error("oh no") end, err_handler)
print("string err", err, type(err))

Result

table err 	table: 67b92f0a 	table
string err 	table: 4f970963 	string

[technomancy]

It looks like a table, but I think it's just because luaj accidentally called tostring on the table and returned that. Either that or the type function is blatantly lying.

[technomancy]

local function err_handler() return {} end

local ok, err = xpcall(function() error({}) end, err_handler)
print("table err", err.gmatch, type(err))

local ok, err = xpcall(function() error("oh no") end, err_handler)
print("string err", err.gmatch, type(err))

table err	nil	table
string err	function: gmatch	string

Looks like a string to me!

[amsitlab]

Its correct with code :

local function f (a,b)
  return a + b
end

local function err (x)
  print ("err called", x)
  return "oh no!"
end

local function pcallfun()
    return f(1,2)
end

status, err, ret = xpcall (pcallfun, err)

print (status)
print (err)
print (ret)

Result

┌[ .../usr/tmp ]─[ ]
└─$ java -cp /storage/emulated/0/Archives/Program/jar/luaj-jse-3.0.2.jar lua scratch2.lua
true
3
nil

┌[ .../usr/tmp ]─[ ]
└─$ lua5.2 scratch2.lua
true
3
nil

[technomancy]

I don't understand what your code is intended to demonstrate. There are no errors involved in that code, so it does not trigger the bug.

My code, which actually does call error, shows the bug because the return value is coerced in a type which is different from what it should be.

Perhaps this would be clearer:

local e = {}
local function err_handler() return e end

local ok, err = xpcall(function() error("oh no") end, err_handler)
print(err == e)

-- $ luaj /tmp/scratch.lua
-- false
-- $ lua5.2 /tmp/scratch.lua
-- true

[amsitlab]

I don't understand what your code is intended to demonstrate. There are no errors involved in that code, so it does not trigger the bug.

My code, which actually does call error, shows the bug because the return value is coerced in a type which is different from what it should be.

Perhaps this would be clearer:

local e = {}
local function err_handler() return e end

local ok, err = xpcall(function() error("oh no") end, err_handler)
print(err == e)

-- $ luaj /tmp/scratch.lua
-- false
-- $ lua5.2 /tmp/scratch.lua
-- true

Maybe this

[SquidDev]

I think (afraid I'm more familiar with LuaJ 2's codebase) the problem occurs here:

luaj/src/core/org/luaj/vm2/LuaClosure.java

Line 570 in daf3da9

le.traceback = errorHook(le.getMessage(), le.level);

errorHook is the function responsible for calling the xpcall callback. It's being called with getMessage rather than getMessageObject, meaning the table is coerced to a string.

In my fork, I found it more convenient to manage the stack unwinding and error function handling within the body of xpcall itself, rather than in the implementation of LuaClosure. But I fear doing anything different is a bit of a breaking change.