Ginkgo linear solver not instantiated in hybrid mode

[pelesh]

The HiOp function, which instantiates linear solver, hiopKKTLinSysCompressedSparseXDYcYd::determineAndCreateLinsys, does not have an option to create Ginkgo linear solver when HiOp hybrid mode is selected (i.e. when linear solver runs on GPU and everything else on CPU). See relevant section of the code.

As a result, when Ginkgo linear solver is selected in hybrid mode, HiOp will hit the assertion checking for GPU-enabled linear solver, and segfault. Ginkgo will happily run on GPU when cpu mode is selected, though.

Workaround: Just run Ginkgo in cpu mode and select CPU or GPU backend in Ginkgo configuration.

Fix: Add Ginkgo stanza to the hybrid mode in determineAndCreateLinsys function.

Question: Do linear solvers need to care about hybrid mode at all? All linear solver cares about is if their data is on the device (GPU) or on the host (CPU). As far as I can see, linear solver does not need to distinguish between cpu and hybrid mode.

CC @nychiang @kswirydo @cnpetra @fritzgoebel @nkoukpaizan

[fritzgoebel]

Am I correct in assuming that no matter if we are in cpu or hybrid mode, the data for the linear solver will in the beginning be on the CPU? If we get the matrix and vectors already on the GPU, that would need some work for the hybrid mode for the Ginkgo solver I think.

[pelesh]

Am I correct in assuming that no matter if we are in cpu or hybrid mode, the data for the linear solver will in the beginning be on the CPU? If we get the matrix and vectors already on the GPU, that would need some work for the hybrid mode for the Ginkgo solver I think.

HiOp has three modes defined: cpu, hybrid and gpu. Only the gpu mode delivers data on the device to the linear solver. The difference between hybrid and cpu mode is that the former tries to select linear solver that runs on the device, if available. The interface for GPU-enabled solver should move the data to the device at each HiOp iteration. I believe this is what Ginkgo interface already does in cpu mode.

[cnpetra]

Just to make sure there is no confusion, cpu mode for HiOp is supposed to run the linear solver on cpu, but I am not opposed of running ginkgo on gpu as long as can run on cpu when a device is not available (or hiop is not built with gpu support). hybrid should run the linear solver on gpu (and, yes, data for the linear solver is going to be on cpu).