Project: LuaRocks new distribution model

LuaRocks new distribution model

Pain point to fix

• users often have a hard time to get LuaRocks up and running;
  • sometimes they end up with multiple installations that mess with each other
  • sometimes LuaRocks can't find its own modules
    • even experienced LuaRocks developers bump into those error messages every now and then when working on complex scenarios
    • Of course "if you do everything right, everything works", but the goal is to make the process less error-prone
  • goal: no more big stacktraces with cannot find module luarocks.foo.bar
  • managing the dependencies of LuaRocks itself is hard
    • how to make LuaRocks depend on other modules reliably (and not get in an infinite dependency loop)

Proposed approach

• building all of LuaRocks into one big luarocks/luarocks.exe executable (including main program, all modules and the Lua interpreter).
  • Eventually, we'll also be able to depend on LuaFileSystem by bundling it, and we'll be able to split the LuaRocks codebase in separete parts (luarocks.fs is a great candidate, and it could also be greatly simplified) and bundle them and maintain them separately from the LuaRocks core.
  • The single executable is not an "installer" that expands into several Lua files: it remains a single executable.
  • This is the approach @leafo takes with the MoonScript interpreter, if my memory serves me right.
• support adding add extra commands to LuaRocks
  • At startup, LuaRocks will scan each directory of the package.path for files named luarocks/cmd/*.lua
  • for better discoverability, some known-but-not-yet-installed extra commands could be listed by luarocks help by default and be automatically installed on first use

Status

Here is a PoC script that implements it:

https://gist.github.com/hishamhm/6f8723f0fced81b0ff2f88e5217ccc69

Initial impressions:

• @tieske: Packing everything in a single executable seems a big change indeed.
• @hisham_hm: It's a great feeling to be able to move around the luarocks binary to a random directory, run it with LUA_PATH=invalid LUA_CPATH=invalid ./luarocks and see it run properly!

Immediate TODO items in that script are:

• hardcoding the "site config" values, which means that the dreaded site_config.lua file ceases to exist
• Windows support (there's one "gcc" line in the end that needs to be changed)
• static linking — note that it links liblua dynamically
  • @hisham_hm: I also tested it linking liblua.a statically (and LuaJIT) and it works, but it needs of course paths taken care by the bootstrap process.
  • ideally, also link stuff like luafilesystem and friends statically to i so that luarocks.exe on Windows does not depend on a bunch of Unix-utils

Open Questions

• how to do the bootstrapping

  • @hisham_hm: to be honest, I was even thinking of the possibility of bootsrapping this using Autotools (it's ugly but I know how to do it, it adds no dependencies for end-users or distro maintainers, and it's solid — been using that on htop since forever)
  • bootstraping on Linux and on Windows are probably two entirely separate stories

• how to interact with tools like hererocks

• maintainability of install-time options

  • options to fiddle with the install, but then again, if this fixes the issues, the need for that is gone

• whether a LR install is specific to a Lua install or if it is serving for multiple Lua versions?

  • @hisham_hm: I've seen people attempting to run multiple Lua versions from a single LR install, but that is not currently supported out of the box.
    • I'd argue it would be easier to support multiple Lua versions with a single executable, due to less interaction with env variables

• how to add additional components to LR later? (e.g. new commands, etc.)

  • this could be done via a regular rock, because the .lua package loader still works. Once the LuaRocks core is up and running, things get reliable, the problem is finding the core.

• what about luarocks.loader? other programs need to find it

  • There are at least two possible solutions:
    1. same as above, and make luarocks.loader an "extra"
    2. a radical option: making the Lua interpreter bundled in the luarocks executable usable for running Lua programs (as in luarocks run foo.lua or even #!/usr/bin/env luarocks run; then luarocks.loader is in package.loaded)
       • this approach does not solve using luarocks.loader from third-party interpreters such as resty, so solution 1 would still be needed.

• if LuaRocks runs on its bundled interpreter, then how does it know for what Lua version it has to install a rock?

  • for Unix, currently a LR installation is fixed for a Lua version — if we make it use the same version it was bundled with it, it wouldn't be any different (that means three different possible binaries)
  • for Windows, current Windows package ships with Lua5.1 but can be configured to use any Lua; but even shipping three binaries in the Windows zip would be simpler than the current setup
  • We can only use the stock lua binaries shipped with LR for running LR itself
    • Once the rocks_trees are installed and the Lua paths for any third-party Lua interpreter is configured, that third-party Lua interpreter can still be used.