Support C++20 modules #6014
Comments
|
Hi, I am interested on working on this enhancement. Can you suggest any resources I can read upon to better help with this issue? |
@Johan511 Perfect!
I'd suggest watching Daniela Egert's talks first, e.g.: Also, I have started working on this a while back (see: https://github.com/hkaiser/hpx/tree/c++20_modules). But had given up because of problems with the cmake support for modules. I think they have made some progress implementing things, though (https://www.youtube.com/watch?v=5X803cXe02Y). |
|
Thank you @hkaiser . I have gone through the talks you have suggested and also through the commit you have made to the c++_20_modules branch. Apart from the changes made to the cmake files most of the other changes seem to be replacing includes from #include <hpx/abc/abc.h> to #include <hpx/module/____.h> which as of now has the same effect. I hope I have properly understood what is being done. |
Tes, that's roughtly what happens. Note, the <hpx/module/...> files are being generated by cmake.
I'd start with making the build system do the right thing (mostly in cmake/HPX_AddModule.cmake). My assumption was that each of the HPX modules (as listed above) should expose their own C++ module (possibly a module part). Those module parts should be composed into two C++ modules that will be consumed by users: one corresponding to the current hpx_core, the other corresponding to the full hpx binary libraries. Thus, the way forward could be to adapt each of the HPX modules one by one from the bottom up. I don't think that that all of HPX need to be adapted. The other thing I would like to achieve is that HPX should be usable without C++ modules for older compilers or by users that don't want to work with C++ modules (yet). |
|
sure, I will try going through HPX_AddModule.cmake
How are we going to achieve this? won't older compilers throw an error on encountering the export keyword? I was thinking about using #ifdef, #ifndef directives to achieve it but am not sure if its the ideal way. |
The generated module header files will have a |
|
I have gone through HPX_AddModule.cmake, the add_hpx_module function seems to be the one which generates the files included by <hpx/module/...> or at least sets necessary contents of the files in various variables, I haven't been able to find the file ( WRITE ) command which writes to the .hpp file
I believe this the following code deals with importing the core and full binary libraries Can you please guide me on what hpx_core is, and how it is different from the hpx binary libraries.
This function seems to be the one which should generate the required .bmi files, as of now most it its definition is commented out. Do we need to work on it to generate the required .bmi files.
|
It's written here: Lines 156 to 159 in aa8143a
hpx_core.so/hpx_core.dll is one of the HPX binaries generated, essentially from all HPX sub-modules in
Yes.
|
|
I have made some changes to the add_hpx_module_library cmake function, can someone please look into it. Also can we have a separate branch to merge all our changes related to supporting modules into. |
I will, thanks!
What do you mean by that? You already have created a PR from that branch, haven't you? |
|
I am currently trying to merge into STEllAR-GROUP:hpx:master, I was suggesting that we could add module support in a separate branch say STEllAR-GROUP:hpx:modules and finally merge that into master. Also, I am currently working with a clone of the fork you had created to implement module support, I hope that is fine. |
|
@Johan511 Sure, that works. I almost think it might be easier to start over from master and step by step reapply necessary changes, i.e. add module support to the build system, and then starting from the lowest level HPX modules convert those to use C++ modules; one by one. |
I also feel the same, it might be better to branch from the current master and go through with the same/similar changes. There seem to be a lot of merge conflicts to deal with. |
|
@hkaiser do you suggest I fork master and start working on it? where do you think I should start? |
Yes, I'd start with creating a new branch and reapplying the build system changes. Then I'd work on exposing the first two or three HPX modules as C++ modules ( |
|
Sorry for the delay, I have gone through the master branch and tried to figure out to what extent modules have been implemented. There seems to be a cmake file HPX_AddModule.cmake which seems to be aimed at generating the modules. Can you confirm if it works as intended? I still am unsure how we are exporting the components of the module we intend to export. My understanding has been that to export at function foo, we use export int foo (); How do we add this "export" keyword into the required files? Can you suggest an environment to test out if my commits are working as intended? I mostly work with Linux (Ubuntu), do I need to shift to Visual Studio on windows to work on this, I have tried using a VM but the performance was not ideal. |
There is a clash in terminology that needs to be clarified, I believe. On one hand, we have the HPX modules (those are not C++20 modules - yet). The HPX modules are a result of code refactorings performed over the last time to build a non-cyclic hierarchy of code blocks that allow to reason about structure in a better way. Each of the sub-directories in The HPX modules are linked into two binary libraries, all from On the other hand, there are C++20 modules. The initial idea was to convert each of the HPX modules into its own C++20 module. If this is done step by step, from the bottom up, we should be able to modularize (C++20 modules) HPX in smaller pieces and still be able to build the whole thing (i.e. tests should pass after each change). The second part of this idea was that we would like to leave using the C++20 modules optional, so that people could still use HPX without opting into using C++ modules. This should also keep HPX compatible with C++17.
I started experimenting with C++20 modules in my PR. The initial thought was to utilize header units. As each of the HPX modules already has a (generated) module header (e.g. #pragma once
#include <hpx/preprocessor/cat.hpp>
#include <hpx/preprocessor/config.hpp>
#include <hpx/preprocessor/expand.hpp>
#include <hpx/preprocessor/identity.hpp>
#include <hpx/preprocessor/nargs.hpp>
#include <hpx/preprocessor/stringize.hpp>
#include <hpx/preprocessor/strip_parens.hpp>You can find those generated module headers under In my PR, I changed this to look like: #pragma once
#if !defined(HPX_HAVE_CXX20_MODULES)
#include <hpx/preprocessor/cat.hpp>
#include <hpx/preprocessor/expand.hpp>
#include <hpx/preprocessor/identity.hpp>
#include <hpx/preprocessor/nargs.hpp>
#include <hpx/preprocessor/stringize.hpp>
#include <hpx/preprocessor/strip_parens.hpp>
#else
import hpx.core.preprocessor;
#endifIOW, if used with modules, #including These headers should eventually be generated, but in the PR those were added explicitly (manually) (e.g. https://github.com/hkaiser/hpx/blob/c%2B%2B20_modules/libs/core/preprocessor/include/hpx/modules/preprocessor.hpp). This scheme also implies that throughout the whole HPX code base, no reference to any of the separate #include's should be made, but all #include's should refer to the generated module header only (except for headers from inside the same HPX module). Also, each HPX module now has added a special export module hpx.core.preprocessor;
export import <hpx/preprocessor/cat.hpp>;
export import <hpx/preprocessor/expand.hpp>;
export import <hpx/preprocessor/identity.hpp>;
export import <hpx/preprocessor/nargs.hpp>;
export import <hpx/preprocessor/stringize.hpp>;
export import <hpx/preprocessor/strip_parens.hpp>;These files could eventually be generated by cmake as well, but for now I have added those manually.
I have no experience with C++ modules when using gcc or clang. My experiments were using MSVC and Visual Studio. I'm not able to suggest anything here. |
|
I have been experimenting with modules with GNU and LLVM compilers too and there do seem to be a few differences related to how they generate module files. As per my understanding GNU seems to generate only .o files which also act as pre-compiled module files, whereas LLVM seems to not expose its modules in its .o files, we need to separately generate .pcm files In case of GNU I was planning to compile HPX modules into a library using something similar to "add_library(mylib STATIC math.cpp)". I have tried doing it for some smaller examples and linked the libraries together with the final executable, and it has worked as expected. I am trying to do the same for clang. I haven't been able to go through the current build process for HPX nor have worked with large scale projects in the past so I hope to receive feedback on if this would be a suggested way of doing it. A statement I would like to confirm is that, does HPX finally compile all it's code into 2 binary libraries (namely libhpx_core.so and libhpx.so) and the end user is supposed to link these libraries into their project? |
|
Hi @gonidelis, thank you for the resource. One thing I would like to point out is that they suggest using the FILE_SET option which seem to be introduced only in CMAKE 3.25, are we comfortable using recently pushed features? |
Yes. People interested in using C++ modules will have to install recent versions of cmake. By the time the dust has settled this will not be a problem anymore (if it ever was). |
|
BTW, in addition to my comment above. The HPX Modules For all HPX modules that depend on config, the module files should be generated by cmake. For instance: #pragma once
#include <hpx/config.hpp>
#if !defined(HPX_HAVE_CXX20_MODULES)
#if defined(HPX_HAVE_MODULE_CACHE)
#include <hpx/cache/entries/entry.hpp>
#include <hpx/cache/entries/fifo_entry.hpp>
#include <hpx/cache/entries/lfu_entry.hpp>
#include <hpx/cache/entries/lru_entry.hpp>
#include <hpx/cache/entries/size_entry.hpp>
#include <hpx/cache/local_cache.hpp>
#include <hpx/cache/lru_cache.hpp>
#include <hpx/cache/policies/always.hpp>
#include <hpx/cache/statistics/local_full_statistics.hpp>
#include <hpx/cache/statistics/local_statistics.hpp>
#include <hpx/cache/statistics/no_statistics.hpp>
#endif // HPX_HAVE_MODULE_CACHE
#elif defined(HPX_CORE_EXPORTS)
import hpx.core.cache;
#else
import hpx.core;
#endifThat means, that if C++ modules are not enabled, any file #including |
|
I have tried going through this, I have faced some errors. I am using cmake 3.25 so don't believe that was the issue. I tried cmake on the examples in cmake repository and faced similar issues. This happens during the cmake phase, so I don't believe it depends on the compiler. Any idea about it? Another error I did come across was FILE_SET only supporting HEADER as an option, I am unsure if I patched it accidentally or its just shadowed by the current error. |
|
@Johan511 you might want to watch this video: https://www.youtube.com/watch?v=5X803cXe02Y. There the speaker explains how to enable experimental support for C++ modules (amongst other things). Essentially this:
|
|
I have gone through quite some material and reached out to people working on module support for C++ 20. This is also a good resource the understand CMake support for C++ modules GCC does not support scanning dependencies in its main version.
These are the errors the user would face, there is a patch which has implemented these command-line options but it hasn't worked for me. gcc-help believes there might be a mistake and has advised me to try again in the patch. I will try it on a container and update the same here. GNU-make does not support modules sources
Only Ninja (>1.12) supports it. set(CMAKE_EXPERIMENTAL_CXX_MODULE_CMAKE_API "2182bf5c-ef0d-489a-91da-49dbc3090d2a") set(CMAKE_EXPERIMENTAL_CXX_MODULE_CMAKE_API "3c375311-a3c9-4396-a187-3227ef642046")
|
|
I have managed to get everything setup, and updated/added some cmake files to add c++20 module. To initially test it out, I am trying to generate a module of all the .hpp files in hpx/preprocessor Also in spite of using
How do I fix it? |
the cmake library Line 314 in e877f5d also, look for all mentions of
Did you delete your cmake cache? |
|
ninja_compile_logs.txt There seem to be issues, with simply trying to compile all the tests using the patches meant for module support. HPX compiles with lot of warnings, but tests for HPX don't even compile. I have attached the logs. Unfortunately diff does not work as ninja runs in parallel. Most of the issues seem to be due to __atomic_load and _GLIBCXX_OPERATOR_DELETE. grep for "/home/hhn/makes/gcc-modules-install" may help as that is where I have installed the GCC patches. building tests fails with |
|
The previous issue was with compiling tests, it was fixed with #include , still facing a few dependency scanning issues. I have sent a PR (it doesn't compile as modules don't seem to be getting built/found) to show the changes made. Any idea what might be the issue? As of now I am just hacking around trying to build a single module and get it to compile. Will work towards a cleaner build once I get I good understanding of what to do. |
@hkaiser any idea what the error might be? |
|
Hi @hkaiser , I am new here and want to contribute to hpx. I think this issue will be good to start with, if I am correct then can you please provide me with some resources else can you please guide me from where can I start to contribute to hpx. |
|
Hi @jainl28patel , I don't think this is the best first issue. As of now we aren't completely sure how to proceed with it. This might be a good set of resources to learn how to contribute. |
|
@jainl28patel I'd agree with @Johan511. The issue here is that creating C++ modules requires to have a good understanding of the API to expose. I think we're not quite at a position where we can fully define this. Even if, at least partially, this could be done today (as far as the compatibility with the Standard is concerned). The bigger obstacle for this is the non-uniform support for modules by the different compilers. MSVC support is decent, and I think we have a good understanding of the technical approach to minimize necessary code changes and still being able to use HPX without modules. I have a proof-of-concept implementation of exposing C++ modules for a very small subset of HPX for MSVC, but I'm not sure whether this would work for gcc and/or clang. To summarize, I'd like to suggest leaving this topic alone for a while, at least until gcc and clang have caught up with their support of C++ modules. What I could suggest you looked into however would be to make our implementations of Would that be something interesting to you? |
|
Thanks @Johan511
@hkaiser Yes! I would like to work on this. Could you provide me with some resource or reference for the task. I am a bit new to developing C++ libraries, but I would love to learn this and implement. |
hkaiser has linked the accepted specification how these data structures should be. If you are feeling lost in the code I would suggest the resources linked earlier. Becoming more comfortable with C++ really helped me navigate HPX easily, hkaiser's lectures (point 5 in resources page) can be really help out with it. |
|
We now track hpx modules support progress at https://arewemodulesyet.org/ . Feel free to create a PR if the status changes at https://github.com/kelteseth/arewemodulesyet/ 😊 |
Level 0:
Level 1:
Level 2:
Level 3:
Level 4:
Level 5:
Level 6:
Level 7:
Level 8:
Level 9:
Level 10:
Level 11:
Level 12:
Level 13:
Level 14:
Level 15:
Level 16:
Level 17:
Level 18:
Level 19:
Level 20:
Level 21:
Level 22:
Level 23:
Level 24:
Level 25:
Level 26:
Level 27:
Level 28:
Level 29:
Level 30:
Level 31:
Level 32:
Level 33:
Level 34:
Level 35:
Level 36:
Level 37:
Level 38:
The text was updated successfully, but these errors were encountered: