[BUG] f2py unable to compile extension module if C sources are generated first

[melissawm]

Following the docs/User Guide from f2py, if one wants to generate a .c file with the extension module one must do

$ f2py -m fib1 fib1.f

to generate a fib1module.c file. This works. However, the next step to build the extension module to be importable in Python, according to the docs, is

$ f2py -c fib1module.c

However, this fails for me with the following messages:

$ f2py -c fib1module.c --verbose
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "untitled" sources
build_src: building npy-pkg config files
running build_ext
new_compiler returns <class 'distutils.unixccompiler.UnixCCompiler'>
customize UnixCCompiler
customize UnixCCompiler using build_ext
********************************************************************************
<class 'distutils.unixccompiler.UnixCCompiler'>
preprocessor  = ['gcc', '-pthread', '-B', '/opt/miniconda/envs/numpy-dev/compiler_compat', '-Wl,--sysroot=/', '-E']
compiler      = ['gcc', '-pthread', '-B', '/opt/miniconda/envs/numpy-dev/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes', '-std=c99']
compiler_so   = ['gcc', '-pthread', '-B', '/opt/miniconda/envs/numpy-dev/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes', '-fPIC', '-std=c99']
compiler_cxx  = ['g++', '-pthread', '-B', '/opt/miniconda/envs/numpy-dev/compiler_compat', '-Wl,--sysroot=/']
linker_so     = ['gcc', '-pthread', '-shared', '-B', '/opt/miniconda/envs/numpy-dev/compiler_compat', '-L/opt/miniconda/envs/numpy-dev/lib', '-Wl,-rpath=/opt/miniconda/envs/numpy-dev/lib', '-Wl,--no-as-needed', '-Wl,--sysroot=/']
linker_exe    = ['gcc', '-pthread', '-B', '/opt/miniconda/envs/numpy-dev/compiler_compat', '-Wl,--sysroot=/']
archiver      = ['ar', 'rc']
ranlib        = None
libraries     = []
library_dirs  = []
include_dirs  = ['/opt/miniconda/envs/numpy-dev/include/python3.6m']
********************************************************************************
building 'untitled' extension
compiling C sources
C compiler: gcc -pthread -B /opt/miniconda/envs/numpy-dev/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -std=c99

compile options: '-I/home/melissa/projects/numpy/numpy/core/include -I/opt/miniconda/envs/numpy-dev/include/python3.6m -c'
gcc: fib1module.c
fib1module.c:16:10: fatal error: fortranobject.h: Arquivo ou diretório inexistente
   16 | #include "fortranobject.h"
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.
error: Command "gcc -pthread -B /opt/miniconda/envs/numpy-dev/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -std=c99 -I/home/melissa/projects/numpy/numpy/core/include -I/opt/miniconda/envs/numpy-dev/include/python3.6m -c fib1module.c -o /tmp/tmpyjgu4iq5/fib1module.o -MMD -MF /tmp/tmpyjgu4iq5/fib1module.o.d" failed with exit status 1

Doing

$ f2py -c fib1module.c -I/home/melissa/projects/numpy/numpy/f2py/src -m fib1module

seems to work (this is the location of my fortranobject.h file), but then the module is not importable:

>>> import fib1module
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /home/melissa/projects/numpy/doc/source/f2py/fib1module.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyFortran_Type

TL; DR: Is this a bug, or something we are not supposed to do (generate .c sources and then build the extension module in a separate step)?

>>> import sys, numpy; print(numpy.__version__, sys.version)
1.18.0.dev0+8d217b0 3.6.7 | packaged by conda-forge | (default, Jul  2 2019, 02:18:42) 
[GCC 7.3.0]

[rgommers]

Thanks for the report @melissawm. We're talking about this page right: https://numpy.org/devdocs/f2py/getting-started.html?

I can confirm what you're seeing:

$ f2py -m fib1 fib1.f
$ f2py -c fib1module.c
...
fib1module.c:16:10: fatal error: 'fortranobject.h' file not found
#include "fortranobject.h"

However, the doc page does $ f2py -c -m fib1 fib1.f instead (generate code and compile it in one go), and that works as expected for me. I'm not sure if separating -m and -c was ever supported. It should be possible to make -c-only work, by automatically adding the right include path - probably not hard, but not sure if it's needed.

@pearu any thoughts?

[pearu]

I am not sure which docs say this but the following is wrong:

$ f2py -c fib1module.c

f2py -c ... must be applied either to existing .pyf file (plus the source/object/library files) or one must specify -m <modulename> option (plus the sources/object/library files). f2py -c is not meant to compile the generated C/API extension modules directly (although it is likely able doing that with proper include directory flags and the source/object file inputs).

Use one of the followinig options:

f2py -c -m fib1 fib1.f

or

f2py -m fib1 fib1.f -h fib1.pyf
f2py -c fib1.pyf fib1.f

These examples are minimal in the sense that if anything is skipped in the above f2py command lines then a failure is expected.

[melissawm]

Actually, I'm talking about this page: https://numpy.org/devdocs/f2py/usage.html

There, in items 2 and 3, we have the following: Item 2 is to run f2py without the -m or -c options, which apparently doesn't work. Item 3 says

To build an extension module, use

f2py -c <options> <fortran files>       \
  [[ only: <fortran functions>  : ]     \
   [ skip: <fortran functions>  : ]]... \
  [ <fortran/c source files> ] [ <.o, .a, .so files> ]

If <fortran files> contains a signature file, then a source for an extension module is constructed,
all Fortran and C sources are compiled, and finally all object and library files are linked to the
extension module <modulename>.so which is saved into the current directory.

If <fortran files> does not contain a signature file, then an extension module is constructed by
scanning all Fortran source codes for routine signatures.

From what you said, I understand we're not supposed to use the .c sources directly, right? If we generate a .c file, is it unusable later?

Thanks for the input!

[pearu]

Actually, I'm talking about this page: https://numpy.org/devdocs/f2py/usage.html

This page describes the f2py command-line options and the different modes. It presumes that one understand how Python extension modules can be built, see for instance https://docs.python.org/3.8/extending/index.html

While f2py can be used for creating extension modules that wrap also C functions (contained in .c files) and hence the statement "we're not supposed to use the .c sources directly" is inaccurate.

One should distinguish the .c files that f2py generates and the .c files that contain user implementations of C functions. The .c file that f2py generates is useful only when one needs to have full control of how the extension module is built or for debugging f2py. In all other cases, one does not need to generate the extension module as it will be generated automatically in the build process.

[melissawm]

I'm adding the explanation from @pearu above to the docs, if you have any other ideas let me know. Thank you all for your coments!

Feel free to close the issue (or should I do it only after that PR gets accepted?)

[pearu]

I suggest closing when the PR that tackles this issue is accepted.

[rgommers]

I suggest closing when the PR that tackles this issue is accepted.

+1

@melissawm you can also add closes gh-14960 to one of your commit messages, then the issue gets auto-closed when the PR gets merged.

[rgommers]

TL; DR: Is this a bug, or something we are not supposed to do (generate .c sources and then build the extension module in a separate step)?

I ran into this again when working on the scipy.linalg Meson build, so let me document how this works before I forget it again (example for fblas):

• Step 1: turn fblas.pyf.src file into fblas.pyf file, using numpy.distutils.from_template.py. This step is not needed if you start from a .pyf file.
• Step 2: run python -m numpy.f2py fblas.pyf --quiet. This generates _fblasmodule.c and _fblas-f2pywrappers.f files.
• Step 3: build your Python extension module.This requires the following source files:
  • _fblasmodule.c
  • _fblas-f2pywrappers.f
  • fortranobject.c (located in numpy/f2py/src/ - you also need to add this dir to your include directories)

Obviously numpy.distutils knows how to collect all these source files, but for other build systems it's pretty tricky.

[rgommers]

To make it more annoying: sometimes a *-f2pywrappers.f is generated, and sometimes it's not. This depends on whether the original .f source code is Fortran 77 (in which case no wrappers are needed), or Fortran 90/95 (then Fortran 77 wrappers are generated by default).

Having the number of generated output files depend on file contents rather than build flags or file extension is not a good thing. Not sure what the best way to avoid it is though, short of renaming F77 files from *.f to *.f77 in SciPy (which I don't really want to do).

[rgommers]

At least we should document the above. The closest is https://numpy.org/devdocs/f2py/usage.html#numpy.f2py.run_main, which says "returns a dictionary containing information on generated modules and their dependencies on source files" with an example. I didn't get this on a first read of that whole "Using F2PY" page though.

@pearu if you have any idea on how to make output files predictable based on input files and command without actually running f2py, I'd love to hear it.

[pearu]

@rgommers one option to achieve predictability would be to always generate the .f and .f90 files, even when these turn out to be empty.

Otherwise, FYI, the files are generated whenever the wrapped routines contain Fortran functions plus some other cases like wrapping F90 module routines.

[rgommers]

one option to achieve predictability would be to always generate the .f and .f90 files, even when these turn out to be empty.

That sounds like a good idea. Not urgent, but if it's easy to do at some point during other f2py work, that would be nice.