Documentation generation on my local machine and on github gives different outcomes

[JulesKouatchou]

Hi Peter,

Thank you again for solving Issue #638. With Ford 7.0.6, I can generate the documentation without any problem on my local Mac. However, when my co-worker attempts to do it in github (same code), there are error messages.

https://github.com/GEOS-ESM/MAPL/actions/runs/8601316694/job/23568290998

Do you know if there is any setting we need to have to make it work?

Thanks.

Jules

---

In case you cannot access the above page, I pasted below part of the Github output:

Preprocessing /home/runner/work/MAPL/MAPL/pfio/tests/Test_Attribute.pf
  Parsing files         ━━━━━ 100% 693/… 0:00… 0:0… pfio/tests/Test_Attribute.pf
  Correlating information from different parts of your project...
Warning: Could not identify ancestor module ('mapl3g_GeomManager') of submodule 
'GeomManager_smod' (in 'GeomManager_smod.F90').
         This is usually because Ford hasn't found 'mapl3g_GeomManager' in any 
of the source directories.

Warning: Could not identify ancestor module ('mapl3g_MaplGeom') of submodule 
'MaplGeom_smod' (in 'MaplGeom_smod.F90').
         This is usually because Ford hasn't found 'mapl3g_MaplGeom' in any of 
the source directories.

Warning: Could not find base type ('MpiTestParameter') of derived type 
'ESMF_TestParameter' (in 'ESMF_TestParameter.F90').
         If 'MpiTestParameter' is defined in an external module, you may be able
to tell Ford about its documentation
         with the `extra_mods` setting
Warning: Could not find base type ('AbstractExportExtension') of derived type 
'RegridExtension' (in 'RegridExtension.F90').
         If 'AbstractExportExtension' is defined in an external module, you may 
be able to tell Ford about its documentation
         with the `extra_mods` setting
Warning: Could not find base type ('MpiTestCase') of derived type 
'ESMF_TestCase' (in 'ESMF_TestCase.F90').
         If 'MpiTestCase' is defined in an external module, you may be able to 
tell Ford about its documentation
         with the `extra_mods` setting
Warning: Could not find base type ('ApplicationMode') of derived type 
'ModelMode' (in 'ModelMode.F90').
         If 'ApplicationMode' is defined in an external module, you may be able 
to tell Ford about its documentation
         with the `extra_mods` setting
Warning: Could not find base type ('Regridder') of derived type 'NullRegridder' 
(in 'NullRegridder.F90').
         If 'Regridder' is defined in an external module, you may be able to 
tell Ford about its documentation
         with the `extra_mods` setting
Warning: Could not find base type ('ApplicationMode') of derived type 
'ServerMode' (in 'ServerMode.F90').
         If 'ApplicationMode' is defined in an external module, you may be able 
to tell Ford about its documentation
         with the `extra_mods` setting
Warning: Could not find base type ('StringVector') of derived type 'DirPath' (in
'MAPL_DirPath.F90').
         If 'StringVector' is defined in an external module, you may be able to 
tell Ford about its documentation
         with the `extra_mods` setting
  ...done in 7.093s
  Processing comments   ━━━━━━━━━━━━━━ 100% 113670/113670 0:00:00 0:01:15 buffer
  Creating HTML documentation... done in 0.543s
  Generating graphs     ━━━━━━━━━━━━━━ 100% 19532/19532 0:00:00 0:11:06 zero_pad
Error rendering page '/home/runner/work/MAPL/MAPL/docs/Ford/mapl3-doc/type/couplermeta.html'
Traceback (most recent call last):
⠙ Creating search index           1%   191… 0:23:43 0:00:… type/couplermeta.html
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/ford/output.py", line 351, in html
    return self.render(self.data, self.proj, self.obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/ford/output.py", line 481, in render
    return self.template.render(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/ford/templates/type_page.html", line 3, in top-level template code
    {% extends "base.html" %}
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/ford/templates/base.html", line 128, in top-level template code
    {% block body %}
^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/ford/templates/type_page.html", line 96, in block 'body'
    {{ macros.bound_info(bp) }}
^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/ford/templates/macros.html", line 380, in template
    {{ binding_summary(bind) }}
^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/ford/templates/macros.html", line 298, in template
    {{ proc_line(proc, proto=proto) }}
^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/ford/templates/macros.html", line 463, in template
    {{ deprecated(proc) }}
  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/ford/templates/macros.html", line 170, in template
    {%- if entity | meta('deprecated') -%}
  ^^^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: Unknown entity 'finalize': This likely means an error in parsing, please check that this file compiles with a Fortran compiler

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.2/x64/bin/ford", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/ford/__init__.py", line 489, in run
    main(proj_data, proj_docs)
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/ford/__init__.py", line 473, in main
    docs = ford.output.Documentation(proj_data, proj_docs, project, page_tree)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/ford/output.py", line 248, in __init__
    self.tipue.create_node(page.html, page.loc, page.meta)
                           ^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/ford/output.py", line 353, in html
    raise RuntimeError(
RuntimeError: Error rendering 'couplermeta.html':
  File "GenericCouplerComponent.F90": type "CouplerMeta"
    Unknown entity 'finalize': This likely means an error in parsing, please check that this file compiles with a Fortran compiler
Error: Process completed with exit code 1.

[ZedThree]

Hi Jules, my initial guess is that this is something to do with checking out the code -- Ford can't find some files that are maybe in an external library or git submodule perhaps?

If you try a clean checkout locally, do you see the same issue?

[mathomp4]

@ZedThree Here is the odd thing. If you look at the GitHub Actions output you'll see (for example):

Preprocessing /home/runner/work/MAPL/MAPL/generic3g/couplers/esmf-way/CouplerMetaComponent.F90
Warning: error when preprocessing 
/home/runner/work/MAPL/MAPL/generic3g/couplers/esmf-way/CouplerMetaComponent.F90
:
cpp -traditional-cpp -E -DUSE_MPI=1 -DBUILD_WITH_PFLOGGER=1 
-DBUILD_WITH_EXTDATA2G=1 -DUSE_FLAP=1 -DH5_HAVE_PARALLEL=1 -DTWO_SIDED_COMM=1 
-DMAPL_MODE=1 -I/home/runner/work/MAPL/MAPL/include 
-I/home/runner/work/MAPL/MAPL/gFTL/install/GFTL-1.13/include/v1 
-I/home/runner/work/MAPL/MAPL/gFTL/install/GFTL-1.13/include/v2 
/home/runner/work/MAPL/MAPL/generic3g/couplers/esmf-way/CouplerMetaComponent.F90
/home/runner/work/MAPL/MAPL/generic3g/couplers/esmf-way/CouplerMetaComponent.F90
:2:2: fatal error: Generic.h: No such file or directory
    2 | 
      |  ^          
compilation terminated.

Reverting to unpreprocessed file

However, we have this file excluded, we think:

exclude: **/EsmfRegridder.F90
...
         **/generic3g/couplers/esmf-way/CouplerMetaComponent.F90

Note that we also tried:

         ../../generic3g/couplers/esmf-way/CouplerMetaComponent.F90

and it still didn't exclude it.

Is there perhaps something we are doing wrong to exclude that file? Maybe if we get the syntax right, we can figure out all the other issues...

[ZedThree]

This is actually a bug, the paths in the settings file are supposed to be relative to the setting file, but (at least) the exclude paths get processed relative to where you run ford, and for some reason **/filename doesn't match filename.

For now, a workaround is to just run ford in the same directory as the settings file, that is, run cd docs/Ford; ford mapl3docs-with-remote-esmf.public_private_protected.md. This seems to work for me

---

Some notes for myself:

This is here:

ford/ford/fortran_project.py

Lines 124 to 127 in 8e05e45

	for exclude in settings.exclude:
	src_files = {
	src for src in src_files if not fnmatch(os.path.relpath(src), exclude)
	}

When ford is run at the top level, then:

fnmatch(os.path.relpath(src), exclude)

becomes:

fnmatch('generic3g/couplers/esmf-way/GenericCoupler.F90', '**/generic3g/couplers/esmf-way/GenericCoupler.F90')

which is False -- which feels a little surprising to me! It might be better to switch to glob.glob(exclude, recursive=True) for pruning the file list, but also os.path.relpath needs to have the second argument with the path of the settings file here

[mathomp4]

This is actually a bug, the paths in the settings file are supposed to be relative to the setting file, but (at least) the exclude paths get processed relative to where you run ford, and for some reason **/filename doesn't match filename.

For now, a workaround is to just run ford in the same directory as the settings file, that is, run cd docs/Ford; ford mapl3docs-with-remote-esmf.public_private_protected.md. This seems to work for me

Ah ha! And I bet both @JulesKouatchou and myself always run ford from the docs/Ford directory because of course we would. But my Github Action didn't do this because, well, I got something working and went hands off.

I'll test this soon.

--

A query @ZedThree: assuming this all works, what is the "right" way to exclude a specific filename in one subdir but not another? Is it:

**/dir1/dir2/file

or:

../../dir1/dir2/file

(since we know exactly where things are relative to the ford control file)

Note we probably have to be specific like this with subdirs, because MAPL3 is slowly becoming based more on path so we might have:

generic3g/foo/bar/module.F90
generic3g/hah/lol/module.F90

and maybe hah/lol is currently a stub that ford can't process (because it's not valid Fortran), but foo/bar is real code that it can. (We do not build hah/lol/module.F90 because it's not in our CMakeLists.txt but ford, of course, is not processing the CMake, it's just looking for Fortran!)

[ZedThree]

A query @ZedThree: assuming this all works, what is the "right" way to exclude a specific filename in one subdir but not another? Is it:

The benefit of the ** notation is that you can exclude similar directories no matter where they appear in your project structure.

The second one is perhaps a bit more explicit, with the downside of maybe being more fragile if you ever rearrange your project directories.

[mathomp4]

@ZedThree Makes sense. We'll stick with the **.

And good news! With your workaround--and some more excludes--we have some docs: https://geos-esm.github.io/MAPL/mapl3-doc/

🎉

Now it's time to start enforcing style, etc. as bits are leaking into places. 😄

MAPL

MAPL is a foundation layer of the GEOS architecture, whose original purpose is to supplement the Earth System Modeling Framework (ESMF)