Building SuiteSparse without Fortran?

[Desperado17]

Greetings,

I want to build suitesparse for an Arm64 target using a linaro based toolchain which as far as I know has no complete Fortran compiler, only gcc, g++ etc.. Is building SuiteSparse possible in such an environment?

Regards

[DrTimothyAldenDavis]

Yes. I have a flag NFORTRAN that allows this (but I plan on renaming that to SUITESPARSE_FORTRAN). The only reason I need Fortran is to figure out how to call the BLAS and LAPACK, but that setting can be done with setting variables.

[DrTimothyAldenDavis]

That sounds like it is supposed to be -lblas but if the BLAS is not found then BLAS_LIBRARIES is empty.

Try setting BLAS_LIBRARIES to the empty string.

[Desperado17]

Basically, I took the clapack from here because I needed cmake:
https://github.com/hunter-packages/clapack
This creates all the expected libraries but doesn't provide a FindBLAS.cmake file or anything else that would export BLAS_LIBRARIES when queried.
In order to proceed, I had to manually set BLAS_LIBRARIES to the path of the libblas.so created by clapack in the CMakeLists.txt of the three components of SuiteSparse that use it.
I wonder if there is a more elegant solution to this.

[Desperado17]

Is there a clean way to configure suitesparse for the case where libblas.so is exported by clapack and has no find_package supported files on its own?

[DrTimothyAldenDavis]

Unsure. If you do a regular system install of libblas.so then FibdBLAS might find it.

But still, OpenBLAS in the Arm might also have LAPACK and it would be much faster

[mmuetzel]

See also #626. That issue would probably also resolve your use case.

[DrTimothyAldenDavis]

Yes, you still need a BLAS library and an LAPACK library.  You just don't need a Fortran compiler to call them. You can configure SuiteSparse to call dgemm, DGEMM, dgemm_, or DGEMM_, or other options, with the SUITESPARSE_C_TO_FORTRAN option.

Ideally, you would want to use an optimized BLAS library, not the standard reference BLAS.

It's possible to compile UMFPACK without the BLAS using -DNBLAS (but this option is not exposed in the CMakeLists.txt script. 
CHOLMOD and SPQR require the BLAS.

The BLAS and LAPACK libraries are found by SuiteSparseBLAS.cmake, in the SuiteSparse_config/cmake_modules folder.  They assume that find_package ( BLAS ) and find_package ( LAPACK ) both work.

[DrTimothyAldenDavis]

Can you use these? https://developer.arm.com/documentation/101004/2310/BLAS-Basic-Linear-Algebra-Subprograms

[DrTimothyAldenDavis]

The libblas.so that you've generated from clapack is probably very slow. It would likely be a translation from Fortran to C of the Fortran Reference BLAS. A better solution would be to look for an ARM-optimized BLAS and LAPACK.

[Desperado17]

You mean something like this? https://developer.arm.com/downloads/-/arm-performance-libraries

[DrTimothyAldenDavis]

Yes, definitely use that. Compiling your own LAPACK is fine for performance, but compiling your own BLAS is a very bad idea. It will work but it will be slow.

The best solution is to get a pre-built LAPACK and BLAS package like that one. I would assume it would support the BLA_VENDOR described here: https://cmake.org/cmake/help/latest/module/FindBLAS.html#blas-lapack-vendors .

[DrTimothyAldenDavis]

And if it does, then SuiteSparse will find it:

SuiteSparse/SuiteSparse_config/cmake_modules/SuiteSparseBLAS.cmake

Lines 102 to 110 in a02c53b

	# Look for ARM BLAS with 64-bit integers
	message ( STATUS "Looking for ARM 64-bit BLAS" )
	set ( BLA_VENDOR Arm_ilp64_mp )
	set ( BLA_SIZEOF_INTEGER 8 )
	find_package ( BLAS )
	if ( BLAS_FOUND )
	include ( SuiteSparseBLAS64 )
	return ( )
	endif ( )

SuiteSparse/SuiteSparse_config/cmake_modules/SuiteSparseBLAS.cmake

Lines 168 to 176 in a02c53b

	# Look for ARM BLAS with 32-bit integers
	message ( STATUS "Looking for ARM 32-bit BLAS" )
	set ( BLA_VENDOR Arm_mp )
	set ( BLA_SIZEOF_INTEGER 4 )
	find_package ( BLAS )
	if ( BLAS_FOUND )
	include ( SuiteSparseBLAS32 )
	return ( )
	endif ( )

Note that a parallel BLAS (Arm_mp or Arm_ilp64_mp) is critical for best performance.

[Desperado17]

I will try that and return in case of difficulties tx.

[mmuetzel]

Another option might be OpenBLAS which can build accelerated BLAS and all LAPACK functions without Fortran compiler. But make sure to use a compatible threading model everywhere. I.e., if you'd like to use OpenMP for the SuiteSparse libraries (and you probably should), also build OpenBLAS with OpenMP.

[Desperado17]

I tried the 9.4.0beta5 code. It gives me difficulties with cross-compiling as it produces an exec format error in GraphBLAS for a grb_jitpackage binary. I assume it expects this to be of x64 host format but it is aarch64 format. Is this needed on host or target and can I make it build for host despite cross-compiling?

[ 14%] Generating compressed sources for JIT compiler...
/bin/sh: 1: /PROPRIETARY/dev/build/system/ext-suitesparse-dev/1/workspace/GraphBLAS/JITpackage/grb_jitpackage: Exec format error
make[2]: *** [GraphBLAS/JITpackage/CMakeFiles/grb_jitpackage_source.dir/build.make:287: GraphBLAS/JITpackage/CMakeFiles/grb_jitpackage_source] Error 126

[mmuetzel]

I removed the constructed file, GB_JITpackage.c, from github, and added a .gitignore so it is no longer uploaded to github. I did this because now the construction of this file is much improved with the new JITpackage/CMakeLists.txt, so now I can reduce the churn on github. A tiny change to the course can cause a huge change to this file.

So each time SuiteSparse is built from the original distro, it will need to construct this file.

If you are cross-compiling, the grb_jitpackage program is not needed on the target. It would just be needed on the host, to build the GB_JITpackage.c file.

Oof. I didn't realize that happened.

In that case, cross-building might indeed get a bit more tricky. CMake would need to invoke the compiler for a native target when it builds the generator. But it needs to invoke the compiler for the cross-build target for the rest...
At the moment, we are working with just one single compiler toolchain (the cross-compiler toolchain in that case).

[mmuetzel]

Would it be possible to revert the change that removed GB_JITpackage.c from the repository for 7.4.0?
Or at least make sure that GB_JITpackage.c is included in the source tarball somehow?

Getting that right looks like a major project...

[DrTimothyAldenDavis]

I suppose I could add it into a final distribution and then add back the .gitignore to the dev2 branch. That's a bit awkward, but I see that getting cross-compilation working would be more difficult. So I will do that for the 7.4.0.beta6 that I can post today.

A "git diff" with any change at all to the source code of GraphBLAS would always lead to a huge change to GB_JITpackage.c, which is why I want to add that file to .gitignore and not post it on github. It also as a configured file, GraphBLAS.h, from GraphBLAS.h.in, inside it, so it can change: it has cmake variables that set the version number. The GRAPHBLAS_HAS_CUDA flag can change the GraphBLAS.h file, in particular. I hope to remove that dependency at some point, and it's not currently an issue since CUDA is entirely disabled in GraphBLAS for now anyway.

[mmuetzel]

@Desperado17: I opened #616 that might work when cross-building GraphBLAS with JIT. Could you please check if that works for you?

[Desperado17]

Tested merge request, result is in the comments there.

Also tested beta7 branch. At the moment it complains about Fortran symbols. Don't know why. OpenBlas, Clapack and suitesparse use NFORTRAN or NOFORTRAN flag:

[ 1%] Building C object AMD/CMakeFiles/AMD_static.dir/Source/amd_valid.c.o [ 1%] Building C object CAMD/CMakeFiles/CAMD_static.dir/Source/camd_preprocess.c.o [ 1%] Building C object CAMD/CMakeFiles/CAMD_static.dir/Source/camd_valid.c.o [ 1%] Linking C shared library libcamd.so [ 1%] Built target CAMD [ 1%] Building C object CCOLAMD/CMakeFiles/CCOLAMD.dir/Source/ccolamd.c.o [ 1%] Building C object AMD/CMakeFiles/AMD_static.dir/Source/amd_version.c.o [ 1%] Linking Fortran shared library libamd.so /usr/bin/ld: CMakeFiles/AMD.dir/Source/amd_1.c.o: Relocations in generic ELF (EM: 183) /usr/bin/ld: CMakeFiles/AMD.dir/Source/amd_1.c.o: Relocations in generic ELF (EM: 183) /usr/bin/ld: CMakeFiles/AMD.dir/Source/amd_1.c.o: Relocations in generic ELF (EM: 183) /usr/bin/ld: CMakeFiles/AMD.dir/Source/amd_1.c.o: Relocations in generic ELF (EM: 183) /usr/bin/ld: CMakeFiles/AMD.dir/Source/amd_1.c.o: Relocations in generic ELF (EM: 183) /usr/bin/ld: CMakeFiles/AMD.dir/Source/amd_1.c.o: Relocations in generic ELF (EM: 183) /usr/bin/ld: CMakeFiles/AMD.dir/Source/amd_1.c.o: Relocations in generic ELF (EM: 183) /usr/bin/ld: CMakeFiles/AMD.dir/Source/amd_1.c.o: error adding symbols: file in wrong format collect2: error: ld returned 1 exit status make[2]: *** [AMD/CMakeFiles/AMD.dir/build.make:512: AMD/libamd.so.3.3.0] Error 1 make[1]: *** [CMakeFiles/Makefile2:1949: AMD/CMakeFiles/AMD.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs....

Edit: Ah wait you changed that flag let me try another time.

[Desperado17]

@DrTimothyAldenDavis I will try once it's up, thank you.

In the meantime: I replaced clapack with openblas with NOFORTRAN flag. When building suitesparse with NFORTRAN it still complains about LAPACK_LIBRARIES missing. Can this be avoided?

Regards

[mmuetzel]

How did you configure OpenBLAS? I don't know if it builds with LAPACK functions by default. You might have to configure with -DC_LAPACK=ON and/or with -DBUILD_WITHOUT_LAPACK=OFF (if you are building it with cmake). There are probably similar flags for their Makefile.

Then when building SuiteSparse, add -DBLA_VENDOR="OpenBLAS" to the configuration flags.

[DrTimothyAldenDavis]

SuiteSparse needs both the BLAS and LAPACK libraries. clapack can be used for the latter. SuiteSparse doesn't need Fortran to find those two packages -- that's separate issue.

[DrTimothyAldenDavis]

Clapack is a C version of LAPACK, translated from Fortran to C via f2c, then cleaned up a bit.

It should be fine for SuiteSparse but I am not sure it has a clapackConfig.cmake or Findclapack.cmake. I guess t that was added in the github repo

[Desperado17]

I'm not sure find_package ( LAPACK will find a clapackConfig.cmake or findclapack.cmake. As I said, I have to replace find_package ( LAPACK with find_package ( CLAPACK for it to be found.

[mmuetzel]

Hmm... I thought that project was called f2clapack...
But maybe there is more than one project that does similar things.

That name reminded me of CBLAS in reference LAPACK:
https://github.com/Reference-LAPACK/lapack/tree/master/CBLAS

[DrTimothyAldenDavis]

Yes, there's a CBLAS but that's different. That's its own API, not a f2c'd version of the BLAS. I would use the CBLAS, since it's a C API, if it were not so stunted. It forces the integer parameters to all be int, so there is no int64_t variant.

[mmuetzel]

In SuiteSparse 7.4.0 and later, you should be able to pass the CLAPACK library as a CMake configuration flag like -DLAPACK_LIBRARIES=/some/path/to/libclapack.so.
You should no longer need to manipulate the .cmake files.

[Desperado17]

Tx!

[DrTimothyAldenDavis]

Are these issues all resolved now?

[DrTimothyAldenDavis]

Resolved in SuiteSparse 7.5.0.

[Desperado17]

Greetings,

I tried 7.5.1 today. That lapack flag seems to work at the moment but a new problem has arised:

find_package ( OpenMP

now fails for GraphBLAS. Any ideas?

[mmuetzel]

Fails how? Which error? Which platform? Which compiler? Which OpenMP? ...

[Desperado17]

I'm cross-compiling for Aarch64 again.

gcc-9.3.0-2.31-edfe925f87-aarch64-linux-gnu

CMake Error at SPEX/CMakeLists.txt:154 (target_link_libraries):
  Error evaluating generator expression:

    $<TARGET_PROPERTY:OpenMP::OpenMP_C,INTERFACE_INCLUDE_DIRECTORIES>

  Target "OpenMP::OpenMP_C" not found.

It does find the FindOpenMP.cmake file provided by the cmake installation but that is not enough.

I tried adding -fopenmp flag to the compiler parameters but then it complains that

aarch64-linux-gnu-gcc: fatal error: cannot read spec file 'libgomp.spec': No such file or directory

Does find_package ( openml need -fopenmp to work?

[mmuetzel]

Correct. If a variable is already in the cache, defaults no longer apply for it. To ignore the cache, you'd need to configure with --fresh.

[mmuetzel]

Ah. I think I now understand parts of your changes. If you start with -DSUITESPARSE_USE_OPENMP=ON, and then configure again with -DSUITESPARSE_USE_OPENMP=OFF, all *_USE_OPENMP flags are overridden with OFF.

However, if you'd then configure again with -DSUITESPARSE_USE_OPENMP=ON (without --fresh), all *_USE_OPENMP would still stay at OFF. That asymmetry is a bit confusing imho...

[DrTimothyAldenDavis]

"That asymmetry is a bit confusing imho..."

Agreed...

The behavior of the cmake cache is confusing to me. Some go in the cache, some don't, and I see no logic as to the difference between the two cases.

[mmuetzel]

As a rule of thumb: All variables that can be changed with -D flags during configuration are cache variables.

As I understand it, the idea behind CMake configuration flags is that they are accumulative. Take, e.g., the following sequence of commands:

cmake -DA=ON -DB=ON -DC=ON ..
cmake -DA=OFF ..
cmake -DC=OFF ..

That would result in a configuration that effectively corresponds to cmake -DA=OFF -DB=ON -DC=OFF ...

That is conceptionally different to autotools configure script flags or Makefile flags (which don't accumulate but always start from a "clean slate").

If you'd like CMake to behave similarly to autotools or Makefiles, you could always add --fresh to the CMake configuration flags in your commands. That essentially ignores the content of the cache (i.e., the content of the CMakeCache.txt file).

Keeping that in mind: Would it be ok to revert that part of the change that overrides the cache in some situations (but doesn't in others)? That would significantly simplify the steps needed to fix this bug.

[DrTimothyAldenDavis]

See #742 for the OPENMP cmake flag issue.

[Desperado17]

SuiteSparse_configConfig.cmake.txt

[Desperado17]

Btw in the meantime I found out that the compiler removed libgomp. But I did not have problems with openmp before so I'm puzzled what's different.

[Desperado17]

ah ok

So what does that mean? That OpenMP was disabled in 9.4.0 beta before?

[mmuetzel]

But note that especially GraphBLAS will probably be much slower if you build it without OpenMP.

[mmuetzel]

I don't know what "before" means for you. But I guess that was before the changes from 038bca2 were released.

I guess your toolchain never contained OpenMP. That was ok prior to that change. SuiteSparse just built without it. But now it ends up with a weird mixture of not having OpenMP but still trying to use it in some places...

[Desperado17]

So if I obtained a compiler that enables libgomp openmp would be enabled? Or do I still have to add -fopenmp manually?

[mmuetzel]

If CMake can detect whatever version of OpenMP you end up installing, you don't need to add that flag. If CMake doesn't detect your version of OpenMP, adding that flag very likely wouldn't make it work automagically without any further flags either.

[Desperado17]

It now proceeds further.

But the native toolchain not set error has returned unless I set the -DGRAPHBLAS_CROSS_TOOLCHAIN_FLAGS_NATIVE variable. :(

[mmuetzel]

But the native toolchain not set error has returned unless I set the -DGRAPHBLAS_CROSS_TOOLCHAIN_FLAGS_NATIVE variable. :(

You'll always need to set that in your cross-building environment. That is not a bug.

[Desperado17]

oh ok

[DrTimothyAldenDavis]

Can this be posted as a new issue or new discussion? It's hard to find otherwise, and the openmp issue has nothing to do with Fortran.

[mmuetzel]

I'm not able to move parts of this discussion to a new topic. But you (as the project maintainer) should be able to move things around afaict.

[DrTimothyAldenDavis]

I can create a new issue or discussion, and I could reference these comments there (one at a time maybe) but I don't know how to move all these to there.

[mmuetzel]

I thought I've seen that in the past. But that was probably the entire discussion that was moved to a different repository.
Moving parts of a discussion doesn't seem to be possible indeed. See this feature request: https://github.com/orgs/community/discussions/2913

[Desperado17]

Another error:

[ 21%] Linking C executable cholmod_di_demo
[ 21%] Building C object GraphBLAS/CMakeFiles/GraphBLAS.dir/Source/FactoryKernels/GB_AxB__any_max_fp64.c.o
/proprietary/dev/dist/toolchain/aarch64-linux-gnu/1/workspace/gcc-9.3.0-2.31-edfe925f87-aarch64-linux-gnu/usr/bin/../lib/gcc/aarch64-linux-gnu/9.3.0/../../../../aarch64-linux-gnu/bin/ld: /proprietary/dev/dist/system/ext-clapack-dev/1/workspace/usr/lib/libf2c.so: undefined reference to `MAIN__'
collect2: error: ld returned 1 exit status
make[2]: *** [CHOLMOD/CMakeFiles/cholmod_di_demo.dir/build.make:103: CHOLMOD/cholmod_di_demo] Error 1

[Desperado17]

Will -DLAPACK_LIBRARIES accept static version liblapack.a ?

[mmuetzel]

Will -DLAPACK_LIBRARIES accept static version liblapack.a ?

I don't know why it wouldn't. But make sure to include all dependencies of that static library in LAPACK_LIBRARIES (if it has any).

[Desperado17]

How are multiple entries separated? lib.a:lib2.a:lib3.a ? Or semicolon?

[Desperado17]

I reconfigured clapack to produce static libraries: liblapack.a, libblas.a, libf2c.a . Now the cholmod test produces linker errors due to symbols that are apparently defined in libf2c.a . I can tell suitesparse the locations of lapack and blas via the environment variables but it is apparently ignorant of the existance of libf2c.a . It only works if I readd the find_package ( CLAPACK call so all necessary library paths are set by the FindCLAPACK.cmake file.

[mmuetzel]

You should be able to either pass a list of libraries separated by semicolon ; or just pass a string that the linker would understand. E.g., -DLAPACK_LIBRARIES="/proprietary/dev/dist/system/ext-clapack-dev/1/workspace/usr/lib/libclapack.a;/proprietary/dev/dist/system/ext-clapack-dev/1/workspace/usr/lib/libtmg.a;/proprietary/dev/dist/system/ext-clapack-dev/1/workspace/usr/lib/libf2c.a" or -DLAPACK_LIBRARIES="-L/proprietary/dev/dist/system/ext-clapack-dev/1/workspace/usr/lib -lclapack -lcblas -ltmg -lf2c" (or whatever is the correct list of library dependencies for CLAPACK).

[DrTimothyAldenDavis]

I've started a new issue regarding the cmake *USE_OPENMP flags, for that part of the discussion. I'll reopen this discussion for the Fortran / LAPACK / BLAS questions.

[Desperado17]

Ok, next problem: I try to build Ceres solver 1.9.0 agains suitesparse 7.5.0:

[ 22%] Building CXX object internal/ceres/CMakeFiles/ceres.dir/dense_qr_solver.cc.o
/proprietary/dev/build/system/ext-okvis-dev/1/workspace/ceres/src/ceres_external/internal/ceres/covariance_impl.cc: In member function 'bool ceres::internal::CovarianceImpl::ComputeCovarianceValuesUsingSparseQR()':
/proprietary/dev/build/system/ext-okvis-dev/1/workspace/ceres/src/ceres_external/internal/ceres/covariance_impl.cc:690:32: error: no matching function for call to 'SuiteSparseQR<double>(int, int, size_t&, cholmod_sparse*, cholmod_sparse**, int64_t**, cholmod_common*)'
  690 |                             &cc);
      |                                ^
In file included from /proprietary/dev/build/system/ext-okvis-dev/1/workspace/ceres/src/ceres_external/internal/ceres/suitesparse.h:49,
                 from /proprietary/dev/build/system/ext-okvis-dev/1/workspace/ceres/src/ceres_external/internal/ceres/covariance_impl.h:41,
                 from /proprietary/dev/build/system/ext-okvis-dev/1/workspace/ceres/src/ceres_external/internal/ceres/covariance_impl.cc:31:
/proprietary/dev/dist/system/ext-suitesparse-dev/1/workspace/usr/include/suitesparse/SuiteSparseQR.hpp:377:55: note: candidate: 'template<class Entry, class Int> Int SuiteSparseQR(int, double, Int, int, cholmod_sparse*, cholmod_sparse*, cholmod_dense*, cholmod_sparse**, cholmod_dense**, cholmod_sparse**, Int**, cholmod_sparse**, Int**, cholmod_dense**, cholmod_common*)'
  377 | template <typename Entry, typename Int = int64_t> Int SuiteSparseQR

Does this version of SuitesparseQR still exist in the code?

[mmuetzel]

Maybe a bug in the code of that older version? Try with the latest version of Ceres Solver 2.2.0.

[mmuetzel]

Probably, this upstream change fixed it for them:
ceres-solver/ceres-solver@9cca671

[Desperado17]

So far it builds with this one and ceres-solver/ceres-solver@d500caa