Skip to content

Fedora compiler environment

Alistair Adcroft edited this page Aug 19, 2022 · 18 revisions

Supplementary instructions for Fedora Linux environments

Noted differences between Fedora Core (FC) 21 and Fedora Core 23. Useful commands to find the software we need to compile MOM6.

  • Package management: FC21: yum, FC23: dnf
  • List files in a package: FC21: rpm -ql [pkg], FC23: dnf repoquery -l [pkg]
  • FC21: pkg-config configuration files are not available for openmpi, see below.

These instructions closely follow the instructions outlined for GFDL's HPC platforms.

Setup a compiler environment on Fedora

Execute the following commands to install the compilation prerequisites on an Fedora machine:

sudo dnf install csh make gcc-gfortran openmpi openmpi-devel netcdf netcdf-devel netcdf-fortran netcdf-fortran-devel netcdf-fortran-openmpi netcdf-fortran-openmpi-devel

Other recommended tools to use git (and unzip), X windows via SSH and syntax colored vim (restart shell session: logout and login again):

sudo dnf install git unzip xauth xterm vim-enhanced

Install the MOM6 source tree

You are ready to use git to download and setup the code repository on the AWS instance.

Create a blank env file

The following commands make use of a file build/gnu/env that can contain any local environment changes need for compilation:

mkdir -p build/gnu
echo > build/gnu/env

Example contents of env using bash

# Need openmpi path before all other paths to get correct nc-config for netcdf with openmpi
export PATH=/usr/lib64/openmpi/bin:${PATH}
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:/usr/lib64/openmpi/lib/pkgconfig

Create a copy of the linux-gnu template and modify

For FC21:

cd src/mkmf/templates
cp linux-gnu.mk linux-gnu-fedora21.mk
# The only real difference is supplying include file and library paths instead
# of using pkg-config.  Replace (-) lines with (+) lines.   Use different paths
# if you installed netcdf or openmpi in other locations.  This demonstrates use
# of openmpi over mpich2.
-FFLAGS += $(shell pkg-config --cflags-only-I mpich2-c)
+FFLAGS += -I/usr/include/openmpi-x86_64 -I/usr/include/openmpi-x86_64/

-CFLAGS += $(shell pkg-config --cflags-only-I mpich2-c)
+CFLAGS += -I/usr/include/openmpi-x86_64 -I/usr/include/openmpi-x86_64/
 
-LIBS := $(shell nc-config --flibs) $(shell pkg-config --libs mpich2-f90)
+LIBS := $(shell nc-config --flibs) -L/usr/lib64/openmpi/lib

For FC23:

cd src/mkmf/templates
cp linux-gnu.mk linux-gnu-fedora23.mk
# Since we chose to using openmpi in this example, change references of mpich2-* to ompi-*
# in linux-gnu-fedora23.mk

NOTE: Change the template to linux-gnu-fedora21.mk for FC21 as needed in the following examples.

Building libfms.a

To build the Makefile use:

mkdir -p build/gnu/shared/repro/
(cd build/gnu/shared/repro/; rm -f path_names; \
../../../../src/mkmf/bin/list_paths -l ../../../../src/FMS; \
../../../../src/mkmf/bin/mkmf -t ../../../../src/mkmf/templates/linux-gnu-fedora23.mk -p libfms.a -c "-Duse_libMPI -Duse_netCDF -DSPMD" path_names)

To build the FMS library (libfms.a):

csh/tcsh:

(cd build/gnu/shared/repro/; source ../../env; make NETCDF=3 REPRO=1 FC=mpif90 CC=mpicc libfms.a -j)

bash:

(cd build/gnu/shared/repro/; . ../../env; make NETCDF=3 REPRO=1 FC=mpif90 CC=mpicc libfms.a -j)

Notice the addition of FC=mpif90 and CC=mpicc above, which is necessary to find the MPI header files.

Building the ocean-only MOM6 executable

To create the Makefile for compiling MOM6:

mkdir -p build/gnu/ocean_only/repro/
(cd build/gnu/ocean_only/repro/; rm -f path_names; \
../../../../src/mkmf/bin/list_paths -l ./ ../../../../src/MOM6/{config_src/infra/FMS1,config_src/memory/dynamic_symmetric,config_src/drivers/solo_driver,config_src/external,src/{*,*/*}}/ ; \
../../../../src/mkmf/bin/mkmf -t ../../../../src/mkmf/templates/linux-gnu-fedora23.mk -o '-I../../shared/repro' -p 'MOM6 -L../../shared/repro -lfms' -c "-Duse_libMPI -Duse_netCDF -DSPMD" path_names)

To build the MOM6 executable:

csh/tcsh:

(cd build/gnu/ocean_only/repro/; source ../../env; make NETCDF=3 REPRO=1 FC=mpif90 LD=mpif90 MOM6 -j)

bash:

(cd build/gnu/ocean_only/repro/; . ../../env; make NETCDF=3 REPRO=1 FC=mpif90 LD=mpif90 MOM6 -j)

This time note the addition of LD=mpif90.

Run MOM6 ocean-only

For any example in ocean_only/ that does NOT have an INPUT/ directory, e.g. double_gyre:

csh/tcsh:

cd ocean_only/double_gyre/
mkdir RESTART
(source ./../../build/gnu/env ; mpirun -np 4 ../../build/gnu/ocean_only/repro/MOM6)

bash:

cd ocean_only/double_gyre/
mkdir RESTART
(. ./../../build/gnu/env ; mpirun -np 4 ../../build/gnu/ocean_only/repro/MOM6)

A successful run should look similar and should match output from a run on an Ubuntu 15.10 system.

This was run on an Amazon AWS instance type m4.xlarge.

MOM Day       0.000      0: En 1.424361E-13, MaxCFL  0.00000, Mass 5.288178268008E+18
    Total Energy: 4126FC96B8ADCFE7  7.5322736070108123E+05
    Total Mass:   5.2881782680077681E+18, Change:   0.0000000000000000E+00 Error:  0.00000E+00 ( 0.0E+00)
MOM Day       1.000     72: En 4.580432E-06, MaxCFL  0.00023, Mass 5.288178268008E+18
    Total Energy: 42B607A80A852EA4  2.4222139843886641E+13
    Total Mass:   5.2881782680077681E+18, Change:  -1.7202096100332227E+01 Error: -1.72021E+01 (-3.3E-18)
MOM Day       2.000    144: En 8.958206E-06, MaxCFL  0.00040, Mass 5.288178268008E+18
    Total Energy: 42C58AE53347EF57  4.7372590026718680E+13
    Total Mass:   5.2881782680077681E+18, Change:  -6.7986416459937118E+01 Error: -6.79864E+01 (-1.3E-17)
MOM Day       3.000    216: En 1.166708E-05, MaxCFL  0.00045, Mass 5.288178268008E+18
    Total Energy: 42CC0E8B5CE0D7C2  6.1697586479535516E+13
    Total Mass:   5.2881782680077681E+18, Change:   1.2942244735786460E+01 Error:  1.29422E+01 ( 2.4E-18)
MOM Day       4.000    288: En 1.542296E-05, MaxCFL  0.00044, Mass 5.288178268008E+18
    Total Energy: 42D28B60D589706D  8.1559337444801703E+13
    Total Mass:   5.2881782680077681E+18, Change:  -1.6385457801725806E+01 Error: -1.63855E+01 (-3.1E-18)
MOM Day       5.000    360: En 1.826022E-05, MaxCFL  0.00053, Mass 5.288178268008E+18
    Total Energy: 42D5F4B97D5D6AA0  9.6563312817578500E+13
    Total Mass:   5.2881782680077681E+18, Change:   3.7420734548125125E+01 Error:  3.74207E+01 ( 7.1E-18)
MOM Day       6.000    432: En 2.397431E-05, MaxCFL  0.00062, Mass 5.288178268008E+18
    Total Energy: 42DCD397A5E1A4E8  1.2678043163201962E+14
    Total Mass:   5.2881782680077681E+18, Change:  -3.8078863737074414E+01 Error: -3.80789E+01 (-7.2E-18)
MOM Day       7.000    504: En 2.927962E-05, MaxCFL  0.00073, Mass 5.288178268008E+18
    Total Energy: 42E19A5109E63388  1.5483585791222025E+14
    Total Mass:   5.2881782680077681E+18, Change:  -3.5114098448941945E+01 Error: -3.51141E+01 (-6.6E-18)
MOM Day       8.000    576: En 3.486441E-05, MaxCFL  0.00081, Mass 5.288178268008E+18
    Total Energy: 42E4F5D995BB638B  1.8436920009398034E+14
    Total Mass:   5.2881782680077681E+18, Change:  -6.0992962031751858E+00 Error: -6.09930E+00 (-1.2E-18)
MOM Day       9.000    648: En 4.195565E-05, MaxCFL  0.00090, Mass 5.288178268008E+18
    Total Energy: 42E9393C9CAC7A8E  2.2186896584802044E+14
    Total Mass:   5.2881782680077681E+18, Change:   5.1411840814233216E+01 Error:  5.14118E+01 ( 9.7E-18)
MOM Day      10.000    720: En 4.604729E-05, MaxCFL  0.00100, Mass 5.288178268008E+18
    Total Energy: 42EBAEF71B9AAA47  2.4350627232699422E+14
    Total Mass:   5.2881782680077681E+18, Change:  -1.9384762671898812E+01 Error: -1.93848E+01 (-3.7E-18)
                              20160512 184045.816: Memuse(MB) at Memory HiWaterMark=  2.965E+01  3.172E+01  8.756E-01  3.020E+01
 MPP_DOMAINS_STACK high water mark=        3680

Supplemental instructions on how to start a single instance on the Amazon Web Services (AWS) Elastic Compute Cloud (EC2)

For Fedora Core 23 (FC23), several pre-constructed base images are available from the Community AMIs. The ami name will be different for different compute regions and zones. By searching on "Fedora-Cloud-Base-23" you can see all the different zones and types available. Linux Amazon Machine Images use one of two types of virtualization: paravirtual (PV) or hardware virtual machine (HVM). The main difference between PV and HVM AMIs is the way in which they boot and whether they can take advantage of special hardware extensions (CPU, network, and storage) for better performance. For example, the image ami-20203441 is a HVM image which can take advantage of the special hardware extensions.

Reference: Nic Hannah, http://nh.id.au/run-an-ocean-model-in-30-mins.html

Author note: Once you have an account established with Amazon AWS and are familiar with how to request and login to the instance, it does literally take 30 minutes or less to run the above example.

Clone this wiki locally