Dynamic asset / build system

[mcclure]

An attempted proposal on something I've been thinking about awhile…

Motivation

Imagine the idea of resources which are generated from other resources. Examples might be:

• Textures which in their pure form are very large images, and are downscaled to generate the textures used in the game
• Texture atlases
• When we start supporting Vulkan, we at some point will need to translate a source language like GLSL into .spvs
• [This is a stretch] A package manager might read a list of packages, then download and install the lua files proper

If one were building a system to support these kinds of resources, a couple things to consider might be:

• You probably don't want to check in the generated files to your version control. You probably don't even want them installed into your project directory, because then either they'll clutter your git statuses or you'll have to set up a complicated .gitignore.
• You want to support multiple configurations, and generate different files depending on those configurations. For example, you might want to downscale your textures smaller on Oculus Quest than on Vive. If there is a WebGPU target, the compile-to-spv step might be followed by a compile-to-webgpu-textual step that isn't present on other platforms.

A naive way to get these goals might be to have a "build" script. You could have a script that combines your project directory with an assets directory to create a "built" project directory. But this would be too bad, because this disrupts the normal way you use Lovr. Lovr doesn't normally have a "build step", and that is nice. So a couple more considerations:

• Ideally, your build step is "optional". It should be possible to run out of a project directory, with anything that needs to be built being built "on the fly".
• Ideally, in the "on the fly" solution, your project directory should still be self-contained. That is, the source files that assets are generated from should live inside the project.
• A byproduct of that last point: The build step should be able to delete as well as create files. When you generate a downscaled image, the full-size image should probably be removed.

A solution: build.lua, Generators

My proposal has three parts:

• lovr.build(t, c)

  This is a function like lovr.conf. It passes in two arguments, t, which works like the t argument to lovr.conf, and c, which describes some things about platform and configuration (like, "you're building for oculus quest" "this is a debug copy").

  t has two members to be assigned to by lovr.build, shun and generators. shun is a list of filenames or fileglobs which should be screened out when doing a build; by default, it contains "build.lua". t.generators is the main purpose of lovr.build, and it is a list of…

• Generators

  These could also maybe be called "converters". I dunno. A generator is a table of the form

  {
    pattern = "FILEGLOB" -- For example: "**.spv", or textures/downscale/*.png"
    generate = function(filename) ... end -- returns 2 values.
                                          -- return 1 is a string or blob containing the contents of the
                                          -- "generated" file.
                                          -- return 2 is optional, and is a path or list of paths
                                          -- these paths are added to the shun list 
  }
  

• build.lua

  Similar to conf.lua, build.lua is a file expected to contain just the definition for function lovr.build.

What these parts do would depend on which mode you're in:

• In regular / just-in-time mode, Lovr runs like Lovr normally runs, but filesystem operations are overloaded. Any file which is either opened or checked for presence is checked against (1) the shun list; if the file is on the shun list, lovr pretends the file doesn't exist and returns an error if you try to open it (2) the generator list; if the file is on the generator list filesystem.isFile advertises its existence, and when you try to read() it, it calls the appropriate generate function with the filename and returns the generate result as the file contents. In addition, we change the execution order: Instead of (conf.lua, lovr.conf, load modules, main.lua, lovr.load), it runs (conf.lua, lovr.conf, load modules, build.lua, main.lua, lovr.build, lovr.load).

• In "build" mode, lovr --build destdir or something, It runs conf.lua then build.lua (but not main.lua). The modules requested in conf.lua are run but the window is not created. Once lovr.build has run and it has the generator and shun lists, it copies the project directory into destdir, and on the way it omits any files on the shunlist and generates all the files on the generator list. lovr.load, etc are not run.

  The result is a directory with the generators "pre-run". No shunned files are present, all dynamically generated files are already generated, and generators are not registered at runtime (because they're registered in build.lua-- which we shunned).

But wait Andi, how does --build mode know what the list of files in the generator list is if the generator list can contain fileglobs

Uh... okay. So I have a couple ideas about that. One is that lovr.build t can contain an explicit list of filenames to generate-- I'm not crazy about this because it could lead to broken builds if your explicit list is missing things. My other two ideas are below.

Stretch goals

There are a couple of more-advanced versions of this idea we could consider.

• Sample generators / Generator generators

  Most projects will probably not have very special generators and will do a small number of obvious things (like: downsample images, build SPIRV). We could offer a small library that generates the most obvious generators.

• Cacheing / Dependencies

  So the idea is every time you read from a file, its contents are generated on the fly. Every time? That's a little wasteful.

  The generator generate function returns a value to be added to the shunlist. We could treat this as a "dependency". When running in normal lovr run mode, it could save a list of all shun-requested files and a lookaside directory with all generated files. Then when it comes time to run the generator, it could check the lookaside and only generate the files if the dependency's datestamp is newer than the last-generated version.

  We could also use a previously recorded list of generated files to generate the list of "all files" for build mode. But there are a lot of ways that could go wrong so I don't like it.

• Functional interface

  The interface for the generators feels pretty complicated and it honestly doesn't seem very "lovr-like". One possibility would be that instead of building up a table, it could call a series of functions-- like, buildSystem.shun("notes/*.txt") or buildSystem.generateForEach({"shaders/*.frag", "shaders/*.vert"}, buildSystem.buildSpv).

  It would be probably easier to make a system like this expressive than the "complicated table of tables of tables" approach so this interface could along the way gather enough information about the files being generated to drive the "make a list of all files" in the --build mode. For example instead of the direction of file autogeneration being "describe a fileglob of files to be built on demand" it could be "in lovr.build scan the directory tree for dynamic-generate sources and register an explicit list of all dynamic-generate files". This would be potentially slower than the purely lazy approach but surely not much since we wouldn't have to run the generators themselves immediately.

Could this be a third-party library?

Historically bjorn has resisted the addition of complicated features to core lovr; this is a complicated feature with a complicated interface. I think lovr is eventually going to need something like this if for no other reason than that when we move to Vulkan you will have to make the .spvs somehow.

However, this entire Thing could in principle be done as a third party library in pure lua, like lovr-lodr.

There are two things that might require this to be an extension to lovr itself rather than a third party library:

1. All lovr functions that load files must lie. lovr.filesystem, as well as any function outside lovr.filesystem that accepts a path as an argument, has to know about the shun and generate lists.

2. Ideally the magic dynamic file loading behavior would work across all threads. (I do the bulk of my file loading off thread.)

I can solve problem (1) by "wrapping" the Lovr API functions; I think ?? my lodr-like tool could just alter the lovr.filesystem table and replace all functions that hit disk with dummy versions that do the generator magic and then call the original functions. However this would be a somewhat laborious process, it wouldn't be forward-compatible (ie every time you released a new lovr version it would break because the function API would change) and it wouldn't propagate to threads because "lodr-type" tools can only impact the main thread scope.

I can solve problems (1) and (2) by introducing my own filesystem functions, like buildSystem.isFile() and buildSystem.loadFile(), and expecting anyone using my third party build system library to call those wrapper functions instead of lovr.filesystem directly. However this will stop working if you use ANY third party lovr libraries (because the third party lovr libraries won't be using my wrappers), and it will be annoying to make work with functions like lovr.graphics.newTexture.

It may be we could find a compromise, like lovr exposes some sort of interface for a dynamic file system to put in hooks and then the build/dynamic file system is a third party library using those hooks.

[bjornbytes]

I think this is a huge need and I agree with a lot of the motivation.

In my head I sorta split this up into 2+ problems:

• I want to be like lovr --build . and get an exe (+ apk + html + whatever) with lots of control over that process (asset conversion, custom icon, ignore dev assets). Terry's feedback about this still haunts me.
• I want to be like lovr.graphics.newShader('shader') and have it automatically compile to spirv during development
• I want lovr to know how to compile shaders and compress textures, but I don't want those tools anywhere in my final build

Thoughts on build.lua / lovr.build

• In general, storing configuration for builds in a file/callback/table like this seems good. I'd like to think about having it in lovr.conf though, like t.release or t.build, but that's minor.
• The generators/shun would work, but IMO the necessary changes to lovr.filesystem are too intrusive

---

Maybe the virtual filesystem can help? It can be used to shadow files. Imagine if something like this ran:

if not lf.isFile('shader.spv') or lf.getLastModified('shader.spv') < lf.getLastModified('shader.glsl') then
  local glsl = lf.read('shader.glsl')
  local spv = require('compiler').compile(glsl)
  lf.write('shader.spv', spv)
end

If that could somehow run, I would always be able to do lovr.graphics.newShader('shader.spv'). I wouldn't have to change my code for dev vs. release. My project folder wouldn't get cluttered with spv assets. It even caches the operation so iteration time is still good. It would work with the existing filesystem module.

---

The other piece of this I was thinking about was using the new plugin system as a way of including optional functionality. There could be a plugin called lovr-dev or lovr-cli that is included in releases that contains a junk drawer of scripts and asset tools.

• It would be a dll, so it can be deleted or omitted from builds.
• It would probably have a Lua API of "raw" functions like lovr.dev.compileShader, lovr.dev.compressTexture.
• It would have something like lovr.dev.handleArgs(arg) that is maybe called by boot.lua.
  • This could be used to implement lovr --build . and other optional CLI commands.

There could maybe be a piece of lovr.dev that handles the lovr.build / build.lua stuff. lovr.dev could use the generators both A) during development, converting assets for you on the fly and maybe writing them to save dir B) whenever it's creating release builds, to convert assets into their final form.

One thing I like about this is that all of the development stuff can be contained in the plugin. LÖVR doesn't know anything about how this stuff works, aside from doing like require('lovr-dev') in boot.lua. The plugin can be its own separate/optional mini-project that makes this stuff easier. People that don't need a build system or need an extra-fancy build system can delete the dll and be on their way. Because it's a plugin it can do weirder experimental things that wouldn't be as appropriate in core lovr.

Still thinking, will post more later