Lua Local Require

[tooolbox]

I noted that require() is relative to Algernon's root run location, even when the currently active Lua file is somewhere else, such as a data.lua or index.lua in my/webapp/path/.

Because of this, there's currently no good way to do ad-hoc code modularization. Using the code lib feature, or require(), both necessitate installing these things as globals within the system. If I want to just modularize some Lua code within a particular folder/Algernon app, I'm not sure how to do that.

@xyproto what do you think of the concept of making require() relative to the location of the file of Lua code that's being executed? I suppose there would be some scenarios where you want to pull in a global module that you installed, as well...

[tooolbox]

My concept here is to make require() act somewhat like the Node module resolution algorithm, in my limited understanding of it, in that it can check in the active script directory, and then parent directories until it reaches the root serving dir, and then in the dir containing serverconf.lua. If it finds the correct Lua module at any point along that path it pulls it in.

Any thoughts @xyproto ?

For now I will start with implementing this myself as an in-Lua thing.

[xyproto]

This sounds like a good plan, as long as performance and security are kept in mind. 👍 🙂

[tooolbox]

Alright, I can get most of the way there using this:

-- loadscriptdir.lua
local r = require
local pp = package.path
require = function(m)
    package.path = scriptdir() .. "/?.lua;" .. pp
    return r(m)
end

-- serverconf.lua
OnReady(function()
    require("loadscriptdir")
end)

That's perfectly acceptable, loads modules from the directory of the running script, which is what I really wanted. Trick is that gopher-lua's module caching system means once it's required, I can't edit the module. I'll think about that one.

[xyproto]

Is this different now that esbuild is in use in main?

[tooolbox]

Well, esbuild is for JS, whereas this is related to Lua modules.

My little script (to import other Lua files relative to the active script dir rather than Algernon's base dir) did the job but as I said it gets cached even in "development mode". Ideally it would only cache in production mode, but I didn't go down that rabbit hole thoroughly.

[tooolbox]

Stepping back:

1. Lua's system of code modularization is modules via the require() function.
2. Things in the Lua ecosystem operate off of the require() convention.
3. Teal is an example of this: it traces require() calls and analyzes those other modules to determine type signatures, then uses this information to report type errors related to how your code is calling into those modules.
4. A Gopher-Lua VM can require() modules. It caches them for its lifetime (which if you're evaluating a script is milliseconds).
5. From what I can tell, Algernon uses a single long-running Gopher-Lua VM to handle processing of .lua files during request handling.
6. If you're writing various .lua endpoints in your Algernon app, (a.lua, b.lua, c.lua) you may want to modularize some code and move it into a shared file (shared.lua) so it can be called from several endpoints. You probably don't want to move it into your root server directory because it doesn't apply to the whole server, you want it in a particular folder with the rest of your app.
7. Following Lua convention, you attempt to use require("shared") in a.lua, but the Gopher-Lua VM executes this in the context of Algernon's root directory, so it doesn't find the shared.lua which is adjacent to a.lua.
8. Modifying the loader chain does allow you to require() an adjacent file.
9. Then we run into the fact that Gopher-Lua caches required modules, since it's one long-running VM serving each request.

I think this becomes more of a philosophical question rather than a purely technical one.

• If you want to expose a math library or something throughout all of Algernon, require() in your serverconf.lua is clearly the right tool for the job.
• If you want to do some simple code reuse in a particular Algernon app, is require() the right tool? (If you want to use Teal as a typechecker, it's the only thing you can use.)
• If require() is the right tool for code reuse within a particular Algernon app, then the module search needs to start within the current script directory (as covered in point 8 above).
• If require() is the right tool for code reuse within a particular Algernon app, then modules shouldn't cache during development (but should definitely cache in production mode, for performance reasons). Alternatively/additionally, some sort of system could be made that checks if required files have changed and if not, uses the cached version of a module.
• Since, from what I recall from my research, require() can mutate the global environment, I believe that some Lua modules out there do not like being loaded twice within the same VM. Normally this is a non-issue since Lua loader implementations (which expect to just execute a script and terminate) use caching. If Algernon's require() implementation does not cache in development configurations, it's not necessarily a problem: any modules that fall into this category would need to be loaded in serverconf.lua only.

Penny for your thoughts @xyproto

[linkerlin]

Alright, I can get most of the way there using this:

-- loadscriptdir.lua
local r = require
local pp = package.path
require = function(m)
    package.path = scriptdir() .. "/?.lua;" .. pp
    return r(m)
end

-- serverconf.lua
OnReady(function()
    require("loadscriptdir")
end)

That's perfectly acceptable, loads modules from the directory of the running script, which is what I really wanted. Trick is that gopher-lua's module caching system means once it's required, I can't edit the module. I'll think about that one.

added one line

    package.path = serverdir() .. "/lib/?.lua;" .. pp

[tooolbox]

To avoid caching modules:

-- loadscriptdir.lua
local r = require
local pp = package.path
require = function(m)
    package.loaded[m] = nil -- Added this
    package.path = scriptdir() .. "/?.lua;" .. pp
    return r(m)
end

-- serverconf.lua
OnReady(function()
    require("loadscriptdir")
end)

Setting package.loaded.myModule to nil before calling require() will ensure that it loads again. Possibly this could be injected into the Lua environment when the Dev mode/flag is on.

Also, dofile() is an alternative, but certain tools (such as the aforementioned Teal) are only looking for require() calls.

[tooolbox]

...Actually, requiring that script in serverconf.lua does not accomplish it, because of Algernon's Lua State Pool. Monkey-patching the require() function only affects that Lua State and when another one is generated, the original require is in place for it. This is really the same reason I had to load Teal in pool.go.

I'm wondering if there's an existing hook for running Lua when a new Lua State is created, or if one could be added.