possibility of adding a build system that finds dependencies and builds lem with whatever it finds and whatever lisp the builder wants

[frejanordsiek]

The status quo is that the builder uses the Makefile to build whichever target they are interested in and may not find out they have a missing dependency (e.g. qlot, sdl2, ncurses, etc.) until the build is quite far along. Additionally, the use of SBCL is hard coded (possibly because that is the only one supported at the present time) but adding more implementations in the future would make the Makefile unwieldy.

This situation could be improved by first making an executable system in the asdf file for each type of build (e.g. lem/executable-ncurses, lem/executable-sdl2, lem/executable-ncurses-sdl2, etc.).

Then, adding a build system like CMake, Autotools, etc. that checks for the dependencies and lets the user select which lisp they want (out of those that are compatible) and then builds the appropriate system like

(load ".qlot/setup.lisp")
(asdf:operate 'asdf:program-op "lem/executable-ncurses-sdl2")

with whichever lisp implementation is chosen.

Would anyone be interested in having such a build system be part of this project? If so, is there a preference for the build system (CMake, Autotools, etc.)? I could write one and submit a merge request for it (very comfortable with CMake and Autotools, and willing to learn others).

[seanfarley]

I, too, have wanted this (and somewhere have a half-baked branch for it) but I lack the experience that @cxxxr and @fukamachi have in the Common Lisp world. I'm curious to see if something better can exist such that we can live in a world of git clone ... && something-maybe-make run Just Works™️ 😄

[frejanordsiek]

Given how the recent xz backdoor was put in an M4 file and how many fewer people can actually check M4 files and configure scripts (let alone actually check them) and the tendency to include the generated M4 files and configure script in tarballs (don't have to, but then one has to hope that the builder has a compatible Autotools), I am inclined to not use Autotools at this point. CMake seems a more reasonable option at this point. Meson might be as well. Less to check. Easier to detect tampering. And more people know them and thus can actually review any changes to them.

[seanfarley]

Personally (I don't know what other CL devs use), I prefer something over CMake but I guess it depends on how easy it is to integrate Meson or whatever else into the project.

[cxxxr]

I am really sorry for the late reply.
It's hard for me to answer anything as I don't know much about this, but I am not keen on autotools either. (I have a history of deleting that set of files in the past.
Personally, I have been concerned about https://github.com/qitab/bazelisp for some time, but it needs to be investigated anyway.

[kinofsolmorrow]

Maybe lake (make like build utility in Common Lisp) could be of interest here?

[frejanordsiek]

Maybe lake (make like build utility in Common Lisp) could be of interest here?

This seems to do the building, but doesn't have much for finding dependencies and configuring the build accordingly (from a first look, so I could be wrong on that). But it gives the power of lisp it seems, so it could be coded to find dependencies and configuring the build, but it could be a lot of code to do so. Still cool though.

I am really sorry for the late reply. It's hard for me to answer anything as I don't know much about this, but I am not keen on autotools either. (I have a history of deleting that set of files in the past. Personally, I have been concerned about https://github.com/qitab/bazelisp for some time, but it needs to be investigated anyway.

Didn't know about bazelisp. Going to look into it. SBCL only, but that is a solvable problem. Most of the lisp code would probably stay the same for other implementations, so it might not be that hard to add support for others.

I've started a proof of concept in CMake since a lot of how to actually build Lem in a separate build directory is the pretty much the same regardless of build system (the incantations differ, but the lisp code that has to run is the same) and to figure out the pain points. Also, SDL2's libraries export CMake find_package modules, so less code to write for the proof of concept. I've already found more than a few of the pain points and some of the other things:

• Getting Qlot to work in the build directory is just a matter of copying qfile and qfile.lock and running qlot from inside that directory.
• Many dependencies are not installed by qlot and must be installed by using the quicklisp that qlot provides. This must be done as a separate build step from making the executable so that the executable is clean-er.
• Have to manually set the ASDF source registry with (asdf:initialize-source-registry ...) just like when (lem:init-at-build-time) is run in the existing build scripts.
• Despite running in the build directory, ASDF dumps the final lem executable in the source directory. This is going to require doing some (asdf:initialize-output-translations ...) to fix. Also, that will let all the FASL files that are built actually be put in the build directory so they can easily be cleaned up.
• It pays off a lot to have a lisp file templated by the build system that loads Qlot's setup.lisp and configures ASDF.

So far, I've found a few issues with using CMake which are strikes against it for something better than a proof of concept:

1. It needs to look for a C compiler in order for it to find ncurses. Part of this is could just be that CMake's FindCurses module is just bad, or that it needs to extract things from the library itself. It may or may not be the case that a C compiler is needed for finding the SDL libraries since those packages install their own finding modules.
2. add_custom_target vs. add_custom_command stuff. Need a target to depend on the final executable. This is annoying and would not be obvious to people new-ish to CMake and make the CMake file a little harder to understand.
3. CMake options (provided to the user) are binary ON/OFF, which will not work easily for telling CMake which implementation to use or to automatically select one.

Other than those issues, the CMake for a proof of concept is going rather well. It turns out to be fairly easy to make it possible to support multiple CL implementations other than the part of users specifying which one (CMake's being able to reference a variable whose name is built from another variable really saves the day here). It is definitely a lot simpler than some other CMake based build systems I've dealt with elsewhere.

So far, it sits at 354 lines of CMake code including comments and whitespace in both the main file and the FindQlot.cmake and FindSBCL.cmake modules. My estimation is about 70 lines of CMake code per CL implementation supported, including comments and whitespace.

Once I get the (asdf:initialize-output-translations ...) step sorted out and add at least one other CL implementation, the proof of concept will be ready for evaluation to determine whether to continue on the CMake road or switch to something else (e.g. Bazel, Meson, etc.).