How to generate module map for compilation database?

[Genomorf]

Recently i watched a fantastic talk from @dmpolukhin named 2023 LLVM Dev Mtg - Improving clangd document open time with preamble caching.

Dmitry mentioned some unbelievable speed up in reopening time in clangd - from 120s to 3s. I would like to reproduce this result on my own project. But my knowledge in c++ modules is limited. Could you please help me to figure out how to generate c++ module map and inject it into compilation database for clangd?

I guess, module map will be just a file with all headers used in my project. And this file will be injected in compile_commands.json using -fmodule-map-file command. If it's correct the question is how to generate such file.

My current setup is: cmake 3.28, g++ (for build), clangd 18.

[dmpolukhin]

@Genomorf we use Python script that extracts all #include directives from the file, resolves them to real files using include paths from CDB, makes them relative to the directory where you will generate modulemap file like this:

module some_name_not_important {
  header "relative_file_path1"
  header "relative_file_path2"
  ...
  export *
}

Instead of implementing include resolver in a script you can add compiler option -H and run only preprocessor on the file and it will print include graph, files on the first level of the graph is what you need. After that you will need to inject the module and other options see full list on the last backup slide.

Unfortunately, other urgent things get me destructed from implementing cache inside clangd itself but I still plan to work on this.

[Genomorf]

Thank you for the answer! I'm trying to make it work now. Using -H flag in g++ i generated the modulemap with all headers from source files. Then, i changed all headers paths to relative paths. Finally, i injected the map in CDB, using flags from your last slide.
I got module map like this:

module A {
   header "../../src/common/security_utils/src/encryption.hpp"
   header "../../../../../my_custom_toolchain/x86_64-linux-gnu/include/c++/13.1.0/memory"
   header "../../.local/.conan/data/fmt/9.1.0/package/12345/include/fmt/core.h"
...
   export *
}

And a .clangd file with the following options:

CompileFlags:
    Add: [-std=c++23, -D__cpp_concepts=202002L, -frelaxed-template-template-args, -fmodules, -fno-implicit-module-maps, -fmodule-map-file=map.modulemap, -fmodules-cache-path=/my_project/build/.cache/clangd, -fmodules-validate-input-files-content, -Wno-module-import-in-extern-c, -Wno-error, -Xclang -fallow-pcm-with-compiler-errors]

But i'm stuck at this moment. Could you please answer to a couple of questions?

1. Do you extract system headers and past it modulemap? E.g. <string>.
2. Why does clangd not recognize this argument Unknown argument: '-Xclang -fallow-pcm-with-compiler-errors'?

Clangd fails to compile any source file. And could not build module A. I found that it could not find headers, which included in header files from module map. E.g.:

V[18:42:26.062] Ignored diagnostic. ./../../.local/.conan/data/openssl/3.0.11/package/123/include/openssl/err.h:16:11:'openssl/macros.h' file not found  

Also, i have lot's of similar logs like File.hpp: in included file: no member named 'find' in namespace 'std::ranges'; did you mean 'cend'?. Seems like clangd fails to find system headers...

Have you any ideas how i could fix these issues?

[dmpolukhin]

@Genomorf your module map looks something like we have. We do put all headers including system to the modulemap. You need to make sure that header in modulemap follow exactly in the same order as in source. Order is important because some header depends on macro and declarations from previous headers in the source. But you can stop you header earlier and include only first headers. We stop on first non #include or command line in the file. It is especially required if you have any #define in the sources file. -Xclang -fallow-pcm-with-compiler-errors is two command line options that follow. Unknown names identifiers means means that you might skipped some headers in modulemap for example.