INSTALL

                    Finite Element Discretization Library
                                   __
                       _ __ ___   / _|  ___  _ __ ___
                      | '_ ` _ \ | |_  / _ \| '_ ` _ \
                      | | | | | ||  _||  __/| | | | | |
                      |_| |_| |_||_|   \___||_| |_| |_|

                               http://mfem.org

The MFEM library has a serial and an MPI-based parallel version, which largely
share the same code base. The only prerequisite for building the serial version
of MFEM is a (modern) C++ compiler, such as g++. The parallel version of MFEM
requires an MPI C++ compiler, as well as the following external libraries:

- hypre (a library of high-performance preconditioners)
  http://www.llnl.gov/CASC/hypre

- METIS (a family of multilevel partitioning algorithms)
  http://glaros.dtc.umn.edu/gkhome/metis/metis/overview

The library supports two build systems: one based on GNU make, and a second one
based on CMake. Both build systems are described below. Some hints for building
without GNU make or CMake can be found at the end of this file.

In addition to the native build systems, MFEM packages are also available in
the Homebrew/Science, https://github.com/Homebrew/homebrew-science, and the
Spack, https://github.com/LLNL/spack, package managers.


Quick start with GNU make
=========================
Serial build:
   make serial -j 4

Parallel build:
   (download hypre 2.10.0b and METIS 4 from above URLs)
   (build METIS 4 in ../metis-4.0 relative to mfem/)
   (build hypre 2.10.0b in ../hypre-2.10.0b relative to mfem/)
   make parallel -j 4

Example codes (serial/parallel, depending on the build):
   cd examples
   make -j 4

Build everything (library, examples and miniapps) with current configuration:
   make all -j 4

Quick-check the build by running Example 1/1p (optional):
   make check


Quick start with CMake
======================
Serial build:
   mkdir <mfem-build-dir> ; cd <mfem-build-dir>
   cmake <mfem-source-dir>
   make -j 4   (assuming "UNIX Makefiles" generator)

Parallel build:
   (download hypre 2.10.0b and METIS 4 from above URLs)
   (build METIS 4 in ../metis-4.0 relative to mfem/)
   (build hypre 2.10.0b in ../hypre-2.10.0b relative to mfem/)
   mkdir <mfem-build-dir> ; cd <mfem-build-dir>
   cmake <mfem-source-dir> -DMFEM_USE_MPI=YES
   make -j 4

Example codes (serial/parallel, depending on the build):
   make examples -j 4

Build everything (library, examples and miniapps) with current configuration:
   make exec -j 4

Quick-check the build by running Example 1/1p (optional):
   make check


Building with GNU make
======================
The MFEM build system consists of two steps: configuration and compilation.

The configuration step can be used to adjust paths to external libraries,
compilers, flags, etc, similar to "./configure". It is performed by running

   make config [OPTIONS] ...

The OPTIONS are of the form VARIABLE=VALUE. Detailed description of the
configuration options is given below. Alternatively, the options can be
specified with an input file:

   cp config/defaults.mk config/user.mk
   (edit config/user.mk)
   make config

Note that config/user.mk, if present, is loaded after config/defaults.mk and
its path/name can be changed with

   make config USER_CONFIG=<user_config_file>

The build system can be configured to use a separate build directory, for an
out-of-source build. There are two ways to do that: the first one is

   mkdir <mfem-build-dir> ; cd <mfem-build-dir>
   make -f <mfem-source-dir>/makefile config [OPTIONS] ...

The second one is

   cd <mfem-source-dir>
   make BUILD_DIR=<mfem-build-dir> config [OPTIONS] ...

Note that in both cases the default location for the (optional) user
configuration file is <mfem-build-dir>/config/user.mk.

Once configured, the library can be built simply with

   cd <mfem-build-dir>   (if building out-of-source)
   make

Note that re-configuration is only needed to change the currently configured
options. Several shortcut targets combining (re-)configuration and compilation
are also defined:

   make serial   -> Builds serial optimized version of the library
   make parallel -> Builds parallel optimized version of the library
   make debug    -> Builds serial debug version of the library
   make pdebug   -> Builds parallel debug version of the library

Note that any of the above shortcuts accept configuration options, either at the
command line or through a user configuration file.

The build can be quick-tested by running

   make check

which will simply compile and run Example 1/1p. For more extensive tests that
check the results from all the serial/parallel MFEM examples and miniapps use:

   make test

Note that by default MFEM uses "mpirun -np" in its test runs (this is also what
is used in the sample runs of its examples and miniapps). The MPI launcher can
be changed by the user as described in the "Specifying a MPI job launcher"
section at the end of this file.

Running all the tests may take a while. Implementation details about the check
and test targets can be found in the top-level makefile and the config/test.mk
file.

An optional installation of the library and the headers can be performed with

   make install [PREFIX=<dir>]

The library will be installed in $(PREFIX)/lib, the headers in
$(PREFIX)/include, and the configuration makefile (config.mk) in $(PREFIX).
The PREFIX option can also be set during configuration.

Information about the current build configuration can be viewed using

   make status
   make info

To clean the library and object files, but keep the current configuration, use

   make clean

To clean everything, including the current configuration, use

   make distclean

For a short help message, use

   make help

The build process creates the MFEM library (libmfem.a) and the include file
(mfem.hpp) needed in MFEM-based applications, see e.g. the example codes in the
examples/ directory or the miniapps in the miniapps/ directory. A selected
subset of configuration options and derived makefile variables are also written
to the file config/config.mk. This file can be included by other makefiles to
obtain information about the MFEM configuration, see e.g. the makefile in the
examples/ directory.


Configuration options (GNU make)
================================
Compilers:
   CXX    - C++ compiler, serial build
   MPICXX - MPI C++ compiler, parallel build

Compiler options:
   OPTIM_FLAGS - Options for optimized build
   DEBUG_FLAGS - Options for debug build
   CXXFLAGS    - If not set, defined based on the above optimized/debug flags
   CPPFLAGS    - Additional compiler options

Installation options:
   PREFIX  - Specify the installation directory. The library (libmfem.a) will be
             installed in $(PREFIX)/lib, the headers in $(PREFIX)/include, and
             the configuration makefile (config.mk) in $(PREFIX).
   INSTALL - Specify the install program, e.g /usr/bin/install

MFEM library features/options (GNU make)
----------------------------------------
MFEM_USE_MPI = YES/NO
   Choose parallel/serial build. The parallel build requires proper setup of the
   HYPRE_* and METIS_* library options, see below.

MFEM_DEBUG = YES/NO
   Choose debug/optimized build. The debug build enables a number of messages
   and consistency checks that may simplify bug-hunting.

MFEM_USE_LIBUNWIND = YES/NO
   Use libunwind to print a stacktrace whenever mfem_error is raised. The
   information printed is enough to determine the line numbers where the
   error originated, provided MFEM_DEBUG=YES or build flags include `-g'.

MFEM_USE_METIS_5 = YES/NO
   Specify the version of the METIS library - 5 (YES) or 4 (NO).

MFEM_USE_LAPACK = YES/NO
   Use LAPACK routines for various dense linear algebra operations. When
   enabled, this option uses the LAPACK_* library options, see below. (When not
   enabled MFEM provides simple internal implementations where appropriate.)

MFEM_THREAD_SAFE = YES/NO
   Use thread-safe implementation for some classes/methods. This comes at the
   cost of extra memory allocation and de-allocation.

MFEM_USE_OPENMP = YES/NO
   Enable (basic) experimental OpenMP support. Requires MFEM_THREAD_SAFE.

MFEM_USE_MEMALLOC = YES/NO
   Internal MFEM option: enable batch allocation for some small objects.
   Recommended value is YES.

MFEM_TIMER_TYPE = 0/1/2/3/4/5/6/NO
   Specify which library functions to use in the class StopWatch used for
   measuring time. The available options are:
      0  - use std::clock from <ctime>, standard C++
      1  - use times from <sys/times.h>
      2  - use high-resolution POSIX clocks (see option POSIX_CLOCKS_LIB)
      3  - use QueryPerformanceCounter from <windows.h>
      4  - use mach_absolute_time from <mach/mach_time.h> + std::clock (Mac)
      5  - use gettimeofday from <sys/time.h>
      6  - use MPI_Wtime from <mpi.h>
      NO - use option 3 if the compiler macro _WIN32 is defined, 0 otherwise

MFEM_USE_SUNDIALS = YES/NO
   Enable MFEM time integrators and non-linear solvers based on the SUNDIALS
   library. When enabled, this option uses the SUNDIALS_* library options,
   see below.

MFEM_USE_MESQUITE = YES/NO
   Enable MFEM functionality based on the Mesquite library. When enabled, this
   option uses the MESQUITE_* library options, see below.

MFEM_USE_SUITESPARSE = YES/NO
   Enable MFEM functionality based on the SuiteSparse library. Currently, this
   option adds the classes UMFPackSolver and KLUSolver (both sparse serial
   direct solvers). When enabled, this option uses the SUITESPARSE_* library
   options, see below.

MFEM_USE_SUPERLU = YES/NO
   Enable MFEM functionality based on the SuperLU_DIST library. Currently, this
   option adds the classes SuperLUSolver (a parallel sparse direct solver) and
   SuperLURowLocMatrix a distributed CSR matrix class needed by SuperLU. When
   enabled, this option uses the SUPERLU_* library options, see below.

MFEM_USE_STRUMPACK = YES/NO
   Enable MFEM functionality based on the STRUMPACK sparse direct solver and
   preconditioner through the STRUMPACKSolver and STRUMPACKRowLocMatrix
   classes. When enabled, this option uses the STRUMPACK_* library options, see
   below.

MFEM_USE_GNUTLS = YES/NO
   Enable secure socket support in class socketstream, using the auxiliary
   GnuTLS_* classes, based on the GnuTLS library. This option may be useful in
   multi-user environment to prevent users from sending/receiving visualization
   data to/from other users. When this option is enabled, the default behavior
   in class socketstream is to use secure sockets, e.g. when connecting to a
   GLVis visualization server. In order for this to work, one needs to generate
   GLVis server/client key pairs (in ~/.config/glvis), similar to ssh keys --
   the script 'glvis-keygen.sh' in the main GLVis directory can be used to do
   that:
      bash glvis-keygen.sh ["Your Name"] ["Your Email"]
   When MFEM_USE_GNUTLS is enabled, the additional build options, GNUTLS_*, are
   also used, see below.

MFEM_USE_NETCDF = YES/NO
   NetCDF is the library that is used by the SNL Cubit mesh generator to create
   Genesis mesh files. This option enables a reader for these files, which
   requires that NetCDF be installed, see the NETCDF_* build options below.

MFEM_USE_PETSC = YES/NO
   Enable MFEM linear and non-linear solvers, preconditioners, time integrators
   and other features based on the PETSc package. When enabled, this option uses
   the PETSC_* library options, see below.

MFEM_USE_MPFR = YES/NO
   MPFR is a library for multiple-precision floating-point computations. This
   option enables the use of MPFR in MFEM, e.g. for precise computation of 1D
   quadrature rules. When enabled, this option uses the MPFR_* library options,
   see below.

MFEM_USE_SIDRE = YES/NO
   Sidre is a component of LLNL's axom project, http://goo.gl/cZyJdn, that
   provides an HDF5-based file format for visualization or restart capability
   following the Conduit (https://github.com/LLNL/conduit) mesh blueprint
   specification. When enabled, this option requires installation of HDF5 (see
   also MFEM_USE_NETCDF), Conduit and LLNL's axom project.

MFEM_USE_GZSTREAM = YES/NO
   Enables use of on-the-fly gzip compressed streams. With this feature enabled
   (YES), MFEM can compress its output files on-the-fly. In addition, it can
   read back files compressed with gzstream (or any compression utility capable
   of creating a gzip-compatible output such as gzip).
   MFEM will write compressed files if the mode argument in the constructor
   includes a 'z' character. With this feature disabled (NO), MFEM will not be
   able to properly read an input file if it is gzip compressed. In that case,
   the solution is to uncompress the file with an external tool (such as gunzip)
   before attempting to use it with MFEM.

MFEM_BUILD_TAG = (any value)
   An optional tag to characterize the build. Exported to config/config.mk.
   Can be used to identify the MFEM build from other makefiles.

VERBOSE = YES/NO
   Print some informational messages when building.

External libraries (GNU make):
------------------------------
Two types of library configuration options are used:
   <LIBNAME>_OPT - for compiler options which usually specify an include path,
                   e.g.: -I/home/user/hypre/include
   <LIBNAME>_LIB - for link options which usually specify link path and library
                   name, e.g.: -L/home/user/hypre/lib -lHYPRE

If specifying relative paths, they should be relative to the top-level MFEM
directory and use the string @MFEM_DIR@, e.g. HYPRE_OPT = -I@MFEM_DIR@/../hypre.

The specific libraries and their options are:

- HYPRE, required for the parallel build, i.e. when MFEM_USE_MPI = YES.
  URL: http://www.llnl.gov/CASC/hypre
  Options: HYPRE_OPT, HYPRE_LIB.

- METIS, required for the parallel build, i.e. when MFEM_USE_MPI = YES. If using
  METIS 5, set MFEM_USE_METIS_5 = YES (default is to use METIS 4).
  URL: http://glaros.dtc.umn.edu/gkhome/metis/metis/overview
  Options: METIS_OPT, METIS_LIB.

- LAPACK (optional), used when MFEM_USE_LAPACK = YES. Alternative, optimized
  implementations can also be used, e.g. the ATLAS project.
  URL: http://www.netlib.org/lapack (LAPACK)
       http://math-atlas.sourceforge.net (ATLAS)
  Options: LAPACK_OPT (currently not used/needed), LAPACK_LIB.

- OpenMP (optional), usually part of compiler, used when MFEM_USE_OPENMP = YES.
  Options: OPENMP_OPT, OPENMP_LIB.

- High-resolution POSIX clocks: when using MFEM_TIMER_TYPE = 2, it may be
  necessary to link with a system library (e.g. librt.so).
  Option: POSIX_CLOCKS_LIB (default = -lrt).

- SUNDIALS (optional), used when MFEM_USE_SUNDIALS = YES.
  In parallel we expect that SUNDIALS is built with support for MPI and hypre.
  URL: http://computation.llnl.gov/projects/sundials/sundials-software
  Options: SUNDIALS_OPT, SUNDIALS_LIB.

- Mesquite (optional), used when MFEM_USE_MESQUITE = YES.
  URL: http://trilinos.org/oldsite/packages/mesquite
  Options: MESQUITE_OPT, MESQUITE_LIB.

- SuiteSparse (optional), used when MFEM_USE_SUITESPARSE = YES.
  URL: http://faculty.cse.tamu.edu/davis/suitesparse.html
  Options: SUITESPARSE_OPT, SUITESPARSE_LIB.

- SuperLU_DIST (optional), used when MFEM_USE_SUPERLU = YES. Note that
  SuperLU_DIST requires ParMETIS, which includes METIS 5 in its distribution.
  Both ParMETIS and the included METIS 5 should be built and installed in the
  same location.
  URL: http://crd-legacy.lbl.gov/~xiaoye/SuperLU
  Options: SUPERLU_OPT, SUPERLU_LIB.

- STRUMPACK (optional), used when MFEM_USE_STRUMPACK = YES. Note that STRUMPACK
  requires the PT-Scotch and Scalapack libraries as well as ParMETIS, which
  includes METIS 5 in its distribution.
  URL: http://portal.nersc.gov/project/sparse/strumpack
  Options: STRUMPACK_OPT, STRUMPACK_LIB.

- GnuTLS (optional), used when MFEM_USE_GNUTLS = YES. On most Linux systems,
  GnuTLS is available as a development package, e.g. gnutls-devel. On Mac OS X,
  one can get the library through the Homebrew package manager (http://brew.sh).
  URL: http://gnutls.org
  Options: GNUTLS_OPT, GNUTLS_LIB.

- NetCDF (optional), used when MFEM_USE_NETCDF = YES, required for reading Cubit
  mesh files. Also requires installation of HDF5 and ZLIB, as explained at the
  NetCDF web site. Note that we use the plain vanilla "C" version of NetCDF, you
  don't need the C++ or parallel versions.
  URL: www.unidata.ucar.edu/software/netcdf
  Options: NETCDF_OPT, NETCDF_LIB.

- PETSc (optional), used when MFEM_USE_PETSC = YES.  Version 3.8 or higher of
  the PETSC dev branch is required. The MFEM and PETSc builds can share common
  libraries, e.g., hypre and SUNDIALS. Here's an example configuration, assuming
  PETSc has been cloned on the same level as mfem and hypre:
    ./configure --download-fblaslapack=yes --download-scalapack=yes \
                --download-mumps=yes --download-suitesparse=yes \
                --with-hypre-dir=../hypre-2.10.0b/src/hypre \
                --with-shared-libraries=0
  URL: https://www.mcs.anl.gov/petsc
  Options: PETSC_OPT, PETSC_LIB.

- Sidre (optional), part of LLNL's axom project, used when MFEM_USE_SIDRE = YES.
  URL: http://goo.gl/cZyJdn (axom, to be released)
       https://github.com/LLNL/conduit (Conduit)
       https://support.hdfgroup.org/HDF5 (HDF5)
  Options: SIDRE_OPT, SIDRE_LIB.

- MPFR (optional), used when MFEM_USE_MPFR = YES.
  URL: http://mpfr.org, it depends on the GMP library: https://gmplib.org
  Options: MPFR_OPT, MPFR_LIB.

- Libunwind (optional), used when MFEM_USE_LIBUNWIND = YES. The library is
  included with OS X (as of version 10.11). On Linux it could be installed with
  the libunwind-devel package.
  URL: http://www.nongnu.org/libunwind
  Options: LIBUNWIND_OPT, LIBUNWIND_LIB.


Building with CMake
===================
The MFEM build system consists of two steps: configuration and compilation.

The configuration step can be used to adjust paths to external libraries,
compilers, flags, etc, similar to any CMake build system. It is performed by
running

   mkdir <mfem-build-dir> ; cd <mfem-build-dir>
   cmake <mfem-source-dir> [OPTIONS] ...

The OPTIONS are of the form -D<VARIABLE>=<VALUE>, e.g. -DMFEM_USE_MPI=YES.
Detailed description of the configuration options is given below. Alternatively,
the options can be specified with an input file:

   cd <mfem-source-dir>/config
   cp defaults.cmake user.cmake
   (edit user.cmake)
   cd <mfem-build-dir>
   cmake <mfem-source-dir>

Note that user.cmake, if present, is loaded before defaults.cmake (and thus the
former takes precedence over the latter) and its path/name can be changed with

   cmake <mfem-source-dir> -DUSER_CONFIG=<user_config_file>

Debug and optimization options are controlled through the CMake variable
CMAKE_BUILD_TYPE which can be set to standard values like "Debug", and "Release"
(default).

To use a specific generator use the "-G <generator>" option of cmake:

   cmake <mfem-source-dir> -G "Xcode"
   cmake <mfem-source-dir> -G "Visual Studio 12 2013"
   cmake <mfem-source-dir> -G "MinGW Makefiles"

With CMake it is possible to build MFEM as a shared library using the standard
CMake option -DBUILD_SHARED_LIBS=1.

Once configured, the library can be built simply with (assuming a UNIX type
system, where the default is to generate "UNIX Makefiles")

   make -j 4
or
   cmake --build .
or
   cmake --build . --config Release  [Visual Studio, Xcode]

The build can be quick-tested by running

   make check
or
   cmake --build . --target check
or
   cmake --build . --config Release --target check  [Visual Studio, Xcode]

which will simply compile and run Example 1/1p. For more extensive tests that
check the results from all the serial/parallel MFEM examples and miniapps use:

   make exec -j 4
   make test
or
   cmake --build . --target exec
   cmake --build . --target test
or
   cmake --build . --config Release --target exec  [Visual Studio, Xcode]
   cmake --build . --config Release --target RUN_TESTS  [Visual Studio, Xcode]

Note that running all the tests may take a while.

Installation prefix can be configured by setting the standard CMake variable
CMAKE_INSTALL_PREFIX. To install the library, use

   make install
or
   cmake --build . --target install
or
   cmake --build . --config Release --target install  [Xcode]
   cmake --build . --config Release --target INSTALL  [Visual Studio]

The library will be installed in <PREFIX>/lib, the headers in <PREFIX>/include,
and the configuration CMake files in <PREFIX>/lib/cmake/mfem.


Configuration variables (CMake)
===============================
Non-standard CMake variables for compilers:
   CXX    - If set, overwrite the auto-detected C++ compiler, serial build
   MPICXX - If set, overwrite the auto-detected MPI C++ compiler, parallel build

The compiler options for the various build types can be controlled using
standard CMake variables like CMAKE_CXX_FLAGS_RELEASE and CMAKE_CXX_FLAGS_DEBUG.

MFEM library features/options (CMake)
-------------------------------------
The following options are equivalent to the GNU make options with the same name:
[see "MFEM library features/options (GNU make)" above]

MFEM_USE_MPI
MFEM_USE_LIBUNWIND
MFEM_USE_LAPACK
MFEM_THREAD_SAFE
MFEM_USE_OPENMP
MFEM_USE_MEMALLOC
MFEM_TIMER_TYPE - Set automatically, can be overwritten.
MFEM_USE_MESQUITE
MFEM_USE_SUITESPARSE
MFEM_USE_SUPERLU
MFEM_USE_STRUMPACK
MFEM_USE_GNUTLS
MFEM_USE_NETCDF
MFEM_USE_MPFR
MFEM_USE_GZSTREAM

The following options are CMake specific:

MFEM_ENABLE_TESTING  - Enable the ctest framework for testing.
MFEM_ENABLE_EXAMPLES - Build all of the examples by default.
MFEM_ENABLE_MINIAPPS - Build all of the miniapps by default.

External libraries (CMake):
---------------------------
For details about the external libraries, see the "External libraries (GNU
make)" section above.

The MFEM CMake build system provides auto-detection for some packages/libraries,
as listed below. The following configuration options are used/defined:
   <LIBNAME>_DIR
      Directory to search for <LIBNAME> first. The exact subdirectories
      searched, for headers and libraries, are chosen based on <LIBNAME>. If the
      library is not found in this location, then standard locations are
      searched.
   <LIBNAME>_REQUIRED_PACKAGES
      Specifies a list of package names that have to be explicitly added when
      linking with <LIBNAME> in addition to its main library, e.g. ParMETIS
      requires METIS, so we set ParMETIS_REQUIRED_PACKAGES to "METIS", see
      defaults.cmake.
   <LIBNAME>_INCLUDE_DIRS
      Location of the <LIBNAME> headers. Set by auto-detection, if successful.
      Can be set explicitly, e.g. if auto-detection fails.
   <LIBNAME>_LIBRARIES
      List of the <LIBNAME> library files/names/link-options. Set by
      auto-detection, if successful. Can be set explicitly, e.g. if
      auto-detection fails.

The CMake build system adds auto-detection for the following packages/libraries:

 - HYPRE
 - METIS - The option MFEM_USE_METIS_5 is auto-detected.
 - MESQUITE
 - SuiteSparse
 - SuperLUDist
 - ParMETIS
 - GNUTLS - Extends the built-in CMake support, to search GNUTLS_DIR as well.
 - NETCDF
 - MPFR
 - LIBUNWIND
 - POSIXCLOCKS

The following built-in CMake packages are also used:

 - MPI, OpenMP, ZLIB
 - LAPACK, BLAS - Both are enabled via MFEM_USE_LAPACK. If auto-detection fails,
      set the <LIBNAME>_LIBRARIES option directly; the configuration option
      <LIBNAME>_DIR is not supported.


Building without GNU make or CMake
==================================
Before using another build system (e.g. Visual Studio) it is necessary to create
a proper configuration header file, config/config.hpp, using the template from
config/config.hpp.in:

   cp config/config.hpp.in config/config.hpp

The file config/config.hpp can then be edited to enable desired options.  The
MFEM library is simply a combination of all object files obtained by compiling
the .cpp source files in the source directories: general, linalg, mesh, and fem.


Specifying a MPI job launcher
=============================
By default, MFEM will use 'mpirun -np #' to launch any of its parallel tests or
miniapps, where # is the number of MPI tasks.  An alternate MPI launcher can be
provided by setting the MFEM_MPIEXEC and MFEM_MPIEXEC_NP config variables.

MFEM will expect the launcher command, plus the command line option to allow it
to specify a number of MPI tasks.

MFEM_MPIEXEC    = mpirun # default
MFEM_MPIEXEC_NP = -np    # default
MFEM_MPIEXEC    = srun   # example for platforms using SLURM
MFEM_MPIEXEC_NP = -n     # example for platforms using SLURM