prototype CMake support

[samuelpmish]

following discussion #1524, this PR aims to investigate feasibility of adding CMake support to libCEED.

At the time of writing, it seems to successfully compile the libCEED static library and tests, although I'm seeing some linker errors on the tests (undefined symbols from unsupported backends), but this is probably because I'm in an unfamiliar code base and don't understand how the backends are supposed to be enabled/disabled.

It also shows an example of generating files (e.g. ceed.pc) when configuring CMake.

Also included is a small example CMake project showing how a downstream user might benefit from CMake integration:

Adding a few lines of code to the CMakeLists.txt makes it possible to download, configure, build and link against libCEED.

[jeremylt]

The logic for checking of a backend should be compiled is here in the Makefile:

libCEED/Makefile

Line 376 in 7ceb642

# Memcheck Backends

[samuelpmish]

tests are compiling now, after adding the missing c sourcefile in backends/. However, the tests are failing with error message

CeedInit(): No resource provided

How should the tests be invoked? Or, is it not worth building tests with the CMake build (if CMake integration exists only for downstream users, who likely won't run the tests)?

[jeremylt]

Tests should be invoked as ./build/t001 /cpu/self/ref/blocked or similar

[jedbrown]

Hi, thanks for putting some effort into implementing. My main concern is that it be relatively low maintenance and relatively reliable. My experience with CMake over the past 15 years has involved a lot of user support working around CMake bugs (SEGV in the CMake executable, infinite loop, subminor release breakage, etc.). Most recently, this was cmake-3.24.2, the default on an HPC machine at the time, hanging forever in select while building a simple package if using MPI compilers or linking to MPI (the package wasn't ours, but since it was a dependency, we got the support email).

If this can actually make things more reliable for users, it would make sense to support. If it means lots of bad user experience and users of other projects coming to use for support because CMake-script that isn't ours is misbehaving, then I'm less enthusiastic.

[jedbrown]

I think we can unify with what is now ceed.pc.template, adopting the CMake syntax if that is required.

[jedbrown]

I think we shouldn't try to innovate here. We could use the same flags as are in the makefile (but hopefully not by copy/pasting them) or (likely preferable) just delete this.

[samuelpmish]

I don't understand, how do you want to mimic the Makefile flags without copying them into the CMake build? Does the Makefile already determine the system hardware and select the appropriate flags automatically?

[jrwrigh]

I think he's specifically referring to the -fvectorize and -fslp-vectorize, which don't appear in our Makefile.

[jedbrown]

Also, the options like -march=native are only applied for gcc/clang, not with vendor compilers for which it isn't allowed. I'd rather have nothing here than default options that break. I also don't love the prospect of manually keeping these compatible, but could be convinced it's worthwhile. One could parse Makefile or otherwise move those defaults to a place that both CMake and the standard Makefile can pull from.

[jedbrown]

Are these undocumented cmake -DCEED_ENABLE_CUDA=1? It fails for me on a vanilla Linux system

$ cmake . -DCEED_ENABLE_CUDA=1
-- The CUDA compiler identification is NVIDIA 12.4.99
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /opt/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Configuring done (1.7s)
-- Generating done (0.0s)
-- Build files have been written to: /home/jed/src/libceed/build-cmake
$ make
[  7%] Building C object CMakeFiles/CEED.dir/interface/ceed-jit-source-root-default.c.o
[  7%] Building C object CMakeFiles/CEED.dir/interface/ceed-basis.c.o
[  7%] Building C object CMakeFiles/CEED.dir/interface/ceed-cuda.c.o
[  7%] Building C object CMakeFiles/CEED.dir/interface/ceed-elemrestriction.c.o
[  7%] Building C object CMakeFiles/CEED.dir/interface/ceed-fortran.c.o
[  8%] Building C object CMakeFiles/CEED.dir/interface/ceed-jit-source-root-install.c.o
[ 11%] Building C object CMakeFiles/CEED.dir/interface/ceed-operator.c.o
[ 11%] Building C object CMakeFiles/CEED.dir/interface/ceed-jit-tools.c.o
In file included from /home/jed/src/libceed/interface/ceed-cuda.c:11:
/home/jed/src/libceed/include/ceed/cuda.h:12:10: fatal error: cuda.h: No such file or directory
   12 | #include <cuda.h>
      |          ^~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/CEED.dir/build.make:90: CMakeFiles/CEED.dir/interface/ceed-cuda.c.o] Error 1
make[2]: *** Waiting for unfinished jobs....
[ 14%] Building C object CMakeFiles/CEED.dir/interface/ceed-preconditioning.c.o
[ 14%] Building C object CMakeFiles/CEED.dir/interface/ceed-qfunction-register.c.o
make[1]: *** [CMakeFiles/Makefile2:100: CMakeFiles/CEED.dir/all] Error 2

[samuelpmish]

I wrote this PR one morning on my laptop, so the initial CUDA bits were sort of just a placeholder. I'm revisiting now on a CUDA workstation, and it seems to be building successfully after a few small changes.

[jedbrown]

Looks like this needs cmake -DCEED_ENABLE_TESTING=1. I get link errors for lack of libm.

/usr/bin/ld/usr/bin/ld: : /usr/bin/ld: ../libCEED.a(ceed-basis.c.o)../libCEED.a(ceed-basis.c.o): in function `: in function `CeedQRFactorization':
ceed-basis.c:../libCEED.a(ceed-basis.c.o)(: in function `.textCeedQRFactorization+0xCeedQRFactorization':
1afc':
ceed-basis.c:)ceed-basis.c:(: undefined reference to `(.textsqrt.text+0x'
+0x1afc/usr/bin/ld1afc): ): undefined reference to `: undefined reference to `../libCEED.a(ceed-basis.c.o)sqrtsqrt: in function `'

It also looks like make -j is producing jumbled output, not sure why.

[samuelpmish]

oops, I rarely use C directly so I didn't realize you had to explicitly link a separate library to use sqrt! I don't know how it managed to compile successfully on my mac, perhaps apple's clang variant links in libm implicitly.

[jedbrown]

Does this run for you? The test executables take the backend as an argument. (Mine fails to link.)

[samuelpmish]

I wasn't able to get the tests to run initially (see #1525 (comment)), but they did compile successfully on my mac and ubuntu workstation.

@jeremylt 's comment about which arguments to pass to the tests was helpful, although for some reason CMake's built in test fixtures pass arguments in quotations, e.g.

test000 "/ref/opt/cuda"

as opposed to

test000 /ref/opt/cuda

which seems to mess up how the tests parse which backends to use.

I'm less familiar with that corner of CMake, so some potential workarounds are to slightly generalize how the tests read the input arguments, or to call the tests through a bash script that strips the extra quotes.

[jedbrown]

How does this pattern work if one wants other libraries to link the same version? E.g., you want PETSc and MFEM to use the same version of libCEED that you might call directly here?

[samuelpmish]

CMake supports a few different options:

• FetchContent (shown here) will both download a dependency and build it as part of the current project.
• ExternalProject will download and build (and/or install) a dependency, but keep it separate from the parent project.
• find_package will look for an installed dependency in the usual system locations for each platform (or wherever you tell it to look)

So, if I had a project that used libCEED directly and also indirectly through mfem

my_project
├── libCEED_version_X.Y.Z
└── mfem
    └── libCEED_version_X.Y.Z

I could either:

1. use the FetchContent to add libCEED to the project and set OVERRIDE_FIND_PACKAGE, so that when mfem uses find_package(libCEED ...) it'll pick up the one I requested.
2. use ExternalProject to build libCEED as well as mfem, and set MFEM_CEED_DIR (or whatever their option is) to the built libCEED
3. have the parent project just write find_package(libCEED ...), and mfem's find_package(libCEED...) will find the same one.

When would I want to use choices 1, 2, or 3 above?

1. if I'm frequently modifying libCEED and am interested in seeing how it affects mfem. If they're built as the same project, then I can modify libCEED sources, and those changes will propagate to mfem and the parent project on the next make.
2. If I just want to automate dependency resolution for the user, but don't anticipate modifying the dependencies.
3. If I already have a version of libCEED installed and just want to use that, rather than building from source.

These aren't necessarily mutually exclusive either. For instance, one could first check for an installed version (3) and if it can't find something, then do either (1) or (2).

[jedbrown]

In the tree above, my_project arguably has global perspective. But what if the dependence on libCEED is in a sibling library to mfem (thus not aware of mfem or my_project. Which of the above options can just work when my_project includes mfem and that sibling library?

[samuelpmish]

if you have

my_project
├── foo
|   └── libbaz_version_X.Y.Z
└── bar
    └── libbaz_version_X.Y.Z

foo and bar can both check if find_package(baz) succeeds, and fallback on FetchContent (for example) if it fails. However, if they don't have that logic in their CMake project, then the parent project needs to be responsible for setting up its subproject's dependencies.

That being said, CMake isn't a package manager. Although it can manage dependencies, for big projects it's often not the right tool for that job. However, if a project has a reasonable CMake implementation, integration with actual package managers like vcpkg and conan is easier.

[jrwrigh]

I guess I'm still not sure why wrapping the Makefile isn't a valid option here? With that, the building logic stays in a central location (our current Makefile) while being able to interface with CMake and it's helper functions.

Like at the bottom of a top-level CMakeLists.txt file, we set a add_custom_target/add_custom_command. Then the rest processes CMake flag requests. For example, if CEED_ENABLE_CUDA, the have what you've already created:

if (CEED_ENABLE_CUDA)
  enable_language(CUDA)
  find_package(CUDAToolkit REQUIRED)
endif()

then we can use the resulting variables to pass CUDA_DIR to make. Repeat that for the other various backend compilers. I've been planning on doing something similar for building against PyTorch, but in reverse (ie. call CMake to run find_package(Torch), then have it print out the required command-line flags).

I think something like this works, while not having to fully support two different build systems.