[cmake-configurator] Support For Metadata

[michelececcacci]

The whole idea of cmake-configurator is to carry metadata that can be specified to give the actual data itself different properties not supported by ccmake. For example ints are not supported, but it would be a better user experience to suport them. The choice would be primarily between some kind of easy to parse meta-data (such as json or yaml), or some easy to read one (like for example a custom one).
Which one would be a better fit? (We should also think about expansibility in the future)

[vvaltchev]

Thanks for opening this conversation, Michele.

I spent some time thinking about this. Let me first describe the problem for everyone. The idea is to develop a cmake configurator similar to ccmake (cmake-curses-gui), but more powerful. Currently, CMake options have a very limited set of types and have no support for groups, dependencies etc.

Now, while it would be possible in theory to design from scratch a new format for describing options and then generating cmake files with a new tool, that would be quite inconvenient because the build system will require a whole new "level of indirection" and CMakeLists.txt files wouldn't be the starting point. The new tool will have to be run at each build to check if the "options metadata" changed and re-generate at least one CMakeLists.txt file. That would look like cumbersome to use and would also be totally incompatible CMake-based build systems. Also, what if we'd like to edit the CMakeLists.txt containing the options? If a tool re-generates that will, we'd loose everything we changed. Therefore, the tool would either have to parse and edit it, or the new format will have to include "echo" statements to emit the rest of the CMake. Therefore, that approach would be like creating a whole pre-processor for CMake, which is already kind-of pre-processor for other build systems (make, ninja etc.). That doesn't look like a good solution to me.

An alternative idea would be to keep CMakeLists.txt untouched, but to keep in parallel some metadata file for extending the capabilities of the CMake options (stored as cached variables). That could work and it will be less invasive than the previous approach, but will go against the "single source of truth" principle (the name of the options would be mentioned in two files without a direct dependency) and that would cause plenty of issues in the long term.

At this point, there only one approach remaining that would keep the CMakeLists.txt files as the "single source of truth" and not require them to be re-generated: put some metadata in the comments. For example, in CMakeLists.txt we have options (cached variables) defined as:

set(KRN_TRACK_NESTED_INTERR ON CACHE BOOL
    "Track the nested interrupts")

set(PANIC_SHOW_STACKTRACE ON CACHE BOOL
    "Show the stacktrace in case of a kernel panic")

set(DEBUG_CHECKS ON CACHE BOOL
    "Keep checking ASSERTs and other debug checks even in release builds")

Those definitions will cause CMake to generate a CMakeCache.txt file in the build directory containing (among many other things):

//Track the nested interrupts
KRN_TRACK_NESTED_INTERR:BOOL=ON

//Show the stacktrace in case of a kernel panic
PANIC_SHOW_STACKTRACE:BOOL=ON

//Keep checking ASSERTs and other debug checks even in release
// builds
DEBUG_CHECKS:BOOL=ON

Tools like cmake-curses-gui just parse that file and allow users to edit the variables with a simple GUI. What if we added some metadata there in the comments? It would be perfect. The only (minor) defect of this approach is that it would be a bit uglier to see such metadata while editing CMakeLists.txt files or why using a tool that doesn't understand it.

Therefore, to reduce this new ugliness, we'll have to try coming up with the format lightweight as possible that will almost "blend in" with the rest of the comment's contents. While a format like JSON would be perfect to parsing, it would be insanely ugly to use while piggybacking data on comments. Imagine this:

set(KRN_TRACK_NESTED_INTERR ON CACHE BOOL
    "Track the nested interrupts { \"group\"=\"debug\", \"depends\"=[] ")

How ugly is that? We'd need a non-invasive format like Markdown, which used things like *bold* that are pleasant to read both with a plain-text editor and with a Markdown-aware editor as well, not something like HTML which terrible to read as source. Therefore, what I'm thinking about currently is a way to put key-value pairs that is as light-weight as possible.

For example, we could have an initial keyword for specifying the beginning of the metadata like [metadata] and then use simple key-value pairs like group=debug; depends_on=a,b,c; type=int. Full-example:

set(KRN_TRACK_NESTED_INTERR ON CACHE BOOL
    "Track the nested interrupts                 [metadata] group=debug; depends_on=DEBUG_CHECKS"

How we'll support string values? We could require them to be in single-quotes 'text' so that it won't require escaping in CMake. E.g.:

set(KRN_TRACK_NESTED_INTERR ON CACHE BOOL
    "Track the nested interrupts                 [metadata] group=debug; depends_on=DEBUG_CHECKS; something='abc'"

But it's clear that we shouldn't see to use strings very often. An alternative syntax that will allow us to skip the initial [metadata] tag would be to wrap each key in [] or {} and claim that comments should never contain those characters. Example:

set(KRN_TRACK_NESTED_INTERR ON CACHE BOOL
    "Track the nested interrupts                 [group] debug; [depends_on] DEBUG_CHECKS, OPT_3; [something] abc"

In this case, even strings won't need quotes, if we continue with the requirement that the special chars won't be used. In theory, even the semi-columns after each option could be skipped with this syntax, e.g.:

set(KRN_TRACK_NESTED_INTERR ON CACHE BOOL
    "Track the nested interrupts                 [group] debug [depends_on] DEBUG_CHECKS, OPT_3 [something] abc"

But I'm not really sure that's so much better even from the human point of view, because it feels unclear when the value finishes. I'm open to further suggestions about the best syntax to use.

[vvaltchev]

Please make some tests with \n etc. I believe last time I checked, ccmake alters the contents of the comments even when no action is taken. So, I believe the trick with \n won't work at all, but might be wrong, of course. No time to check it myself now.

About the syntax, it was to be as light as theoretically possible. The heavy syntax is unpleasant to read and write. Because this is not a general-purpose text format, it should be totally fine to assume that identifiers like DEBUG_CHECKS won't contain any special characters.
I'm open to more ideas, with the goal to make the syntax even nicer, not heavier (less that's strictly necessary of course).

[michelececcacci]

The \n thing works(ccmake doesn't display anything after it), so i think i'm definitely going to implement it.
Just for reference:

set(ARCH_GTESTS         OFF CACHE BOOL   "Build unit tests for the target arch 
dfljlsjfla
fldjasldfjal
lfdjasldfj")

produces this:

//Build unit tests for the target arch 
//\ndfljlsjfla
//\nfldjasldfjal
//\nlfdjasldfj

In regards to the syntax itself, i'm still considering options, but i will keep in mind that the syntax should be as light as possible.

[michelececcacci]

I thought about the language for a bit, and I think this is a great way to do it.

set(KRN_TRACK_NESTED_INTERR ON CACHE BOOL
    "Track the nested interrupts
      group debug
      depends_on DEBUG_CHECKS, OPT_3
      something abc
"

Since cmake normally doesn't split comments in more lines, we could take advantage of this, and simply state that when we find a line that's not the first in the comments, it belongs to a variable. this would be perfect, since we could always take value the first and turn the variable into an array if there is more than one match (very similar to yaml). We also don't need to use something like a semicolon to mark the line end, since we can assume there will just be one "variable" in every line, making the syntax very easy to write.
maybe we could use

set(KRN_TRACK_NESTED_INTERR ON CACHE BOOL
    "Track the nested interrupts
      group=debug
      depends_on=DEBUG_CHECKS, OPT_3
      something=abc
"

[vvaltchev]

Great idea, Michele! Thanks so much for thinking about that!
I liked the first idea a lot:

set(KRN_TRACK_NESTED_INTERR ON CACHE BOOL
    "Track the nested interrupts
      group debug
      depends_on DEBUG_CHECKS, OPT_3
      something abc
"

ccmake doesn't show the 2nd line and the format looks both clean and usable. Great job!
Now it remains just to figure out what's the prettiest coding style for that and to do the implementation of course.

For the moment, I propose the following coding style, even if I'm sure it could be further improved:

set(
   KRN_TRACK_NESTED_INTERR ON CACHE BOOL

   "Track the nested interrupts
   GROUP debug   
   DEPENDS_ON A, B, C"
)

Note the upper-case metadata tags. I guess that forcing them to be upper case could be overall better for the format.

[michelececcacci]

Perfect, i'm going to implement this in the following days!