can't build with external mfem

[jtristano]

My folder structure is:

./mfem
|-libmfem.so.4.6.1
|-libmfem.so (symbolic link ../libmfem.so.4.6.1)
|-lib/libmfem.so ( symbolic link ../libmfem.so)
|- standard mfem github layout

./PyMFEM

I run python3 setup.py install setup.py install --mfem-source=/home/user/mfem --mfem-prefix=/home/user/mfem --skip-ext

And get a failure in cannot find mpi.h

[sshiraiwa]

For MPI.h, we need to specify MPI wrapper so that the compiler knows where MPI.h is located. Here is
an example with intel MPI.

python setup.py install --with-parallel --CC=icc --CXX=icpc --MPICC=mpiicc --MPICXX=mpiicpc

Please let me know this helps

[jtristano]

Thanks! I got a but further... I tried

python3 setup.py install --mfem-source=/home/user/mfem --mfem-prefix=/home/user/mfem --with-parallel --skip-ext --MPICXX=/usr/bin/mpicxx --MPICC=/usr/bin/mpicc --CC=gcc --CXX=gcc --hypre-prefix=/usr/local --metis-prefix=/usr/local

Then loads of no such file mpi.h in globals.hpp

Here's the beginning of the output:

running install
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
----configuration----
 prefix /usr
 when needed, the dependency (mfem/hypre/metis) will be installed under 
 build mfem : No
 build metis : No
 build hypre : No
 build libceed : No
 build gslib : No
 call SWIG wrapper generator: Yes
 build serial wrapper: Yes
 build parallel wrapper : Yes
 hypre prefix /usr/local
 metis prefix /usr/local
 c compiler : gcc
 c++ compiler : gcc
 mpi-c compiler : /usr/bin/mpicc
 mpi-c++ compiler : /usr/bin/mpicxx
 verbose : No
 SWIG : /usr/local/bin/swig

running build
running build_py```




Then failure

[sshiraiwa]

Hi @jtristano
I try a plain installation using Docker python image. A short suggestion, can you add --no-serial together with --with-parallel?

(Long version)
I think what is happening is following. Your mfem installation is build with MFEM_USE_MPI, and thus, "mfem.hpp" includes MPI.h. PyMFEM comes with a wrapper for serial version of MFEM and parallel version of MFEM, and by default, it tries to build the serial version. Since the serial version code is compiled using cc and cc does not know the location of MPI.h, the build fails. --no-serial option should allows you skipping the serial build. Note that you can use import mfem.par, but not import mfem.ser, using this option.

[sshiraiwa]

Closing for now. Feel free to reopen.

[jtristano]

Tried it , got a bit further, had to add --with-cuda as well as --no-serial ,
Now fatal error
gridfunc_wrap.cxx:3933:12: fatal error: 'mfem/mfem.hpp' file not found
#include "mfem/mfem.hpp"

[sshiraiwa]

It is difficult to identify what is the issue with this information. I guess your MFEE is built with
MFEM_USE_CUDA=YES? Here is an example to build PyMFEM linked with MFEM, HYPRE and METIS
built separately, using nvidia/cuda:12.3.1-devel-ubuntu22.04.

### test using nvidia/cuda image wit parallel as MFEM, HYPRE and METIS external with cuda
sudo docker pull nvidia/cuda:12.3.1-devel-ubuntu22.04
sudo docker run  --gpus all -it nvidia/cuda:12.3.1-devel-ubuntu22.04

(inside docker)
apt-get install python3-dev
apt-get install python3-pip
apt-get install libmpich-dev
apt-get install wget
apt-get install git
pip install cmake swig six numba-scipy mpi4p

wget https://github.com/mfem/tpls/raw/gh-pages/metis-5.1.0.tar.gz
wget https://github.com/hypre-space/hypre/archive/v2.28.0.tar.gz

* Hypre
tar -zxvf v2.28.0.tar.gz
mv hypre-2.28.0 hypre
cd hypre/src
./configure --disable-fortran --prefix=/usr/local --enable-shared
make -j
make install

* METIS (after expanding tar-ball)
make config OPTFLAGS=-Wno-error=implicit-function-declaration prefix=/usr/local shared=1
make -j
make install

* MFEM
git clone https://github.com/mfem/mfem.git
cd mfem
make config MFEM_USE_MPI=YES MFEM_USE_CUDA=YES MFEM_USE_METIS_5=YES PREFIX=/usr/local SHARED=YES
make  -j
make install

# PyMFEM
git clone https://github.com/mfem/pyMFEM.git
cd PyMFEM
python3 setup.py install  --mfem-source=../mfem --mfem-prefix=/usr/local --with-parallel --with-cuda --MPICXX=mpicxx --MPICC=mpicc --CC=gcc --CXX=g++ --hypre-prefix=/usr/local --metis-prefix=/usr/local --no-serial
(above installes mfem under site-pakcage dir. somehow, the Python in this docker looks at dist-packages
 for now, the following export works)
export PYTHONPATH=/usr/lib/python3.10/site-packages/:$PYTHONPATH
cd examples
python3.10 ./ex1p.py -pa -d cuda

In general, in order to use externally built MFEM, the MFEM has to built from a certain version/commit. SHA of
MFEM, with which we test and build PyPI distribution is specified at https://github.com/mfem/PyMFEM/blob/master/setup.py#L65. It is not necessary to use the exact same SHA. But, typically, earlier than what is specified
would likely to have build issues.

[jtristano]

Here's the SHA1 we are on for mfem - d5aa18f7625032d35e105381f2d34b4a2ffb2db1

[jtristano]

Got it to build, I hadnt done a make install on mfem...

Is there any way to get both serial and parallel install for PyMFEM with an mfem built with mpi?

[jtristano]

I'm good now ! Thank you for the help... we can close

[sshiraiwa]

Glad that this is resolved

[jtristano]

@sshiraiwa - I'm trying to get this working in our CICD pipeline and found that the build is now failing.

I've pegged the mfem version to your specified sha1 and have used the PyMFEM head

Getting this error

/home/xxx/mfem/include/mfem/fem/bilininteg.hpp:3535: Warning 511: Can't use keyword arguments with overloaded functions (mfem::IdentityInterpolator::AssemblePA(mfem::FiniteElementSpace const &,mfem::FiniteElementSpace const &)).
VectorQuadratureLFIntegrator
    def __init__(self, vqfc, ir):

Traceback (most recent call last):
  File "/home/xxx/PyMFEM/mfem/common/generate_lininteg_ext.py", line 30, in <module>
    assert False, "No recipt for this pattern "
AssertionError: No recipt for this pattern

[jtristano]

swig 4.2 cannot build PyMFEM... max version I can build with is 4.1.1.post1

[sshiraiwa]

Reopening this. (#208 is in progress)