libpostal/libpostal.h: No such file or directory

[michal-ko]

I'm getting these error logs when I try to install postal on Fedora 34. I have installed libpostal ahead of time using:

• git clone https://github.com/openvenues/libpostal
• cd libpostal
• ./bootstrap.sh
• ./configure --datadir=/home/mike/PycharmProjects/libpostal_data
• sudo make -j4
• sudo make install
• sudo ldconfig

I checked after installation it works by running ..libpostal/src/address_parser and I can confirm it does indeed work

I've added to my .bashrc export LD_LIBRARY_PATH=/home/mike/PycharmProjects/libpostal/

Now, if I try to install Python pypostal package I keep getting the following error

`(venv) bash-5.1$ python -m pip install postal
Collecting postal
Using cached postal-1.1.9.tar.gz (19 kB)
Requirement already satisfied: six in /home/mike/PycharmProjects/govpole_prototype/venv/lib/python3.9/site-packages (from postal) (1.16.0)
Building wheels for collected packages: postal
Building wheel for postal (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /home/mike/PycharmProjects/govpole_prototype/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/run/user/1000/app/com.jetbrains.PyCharm-Community/pip-install-wvz_dad4/postal_34f58629c378450382251c29da84f37b/setup.py'"'"'; file='"'"'/run/user/1000/app/com.jetbrains.PyCharm-Community/pip-install-wvz_dad4/postal_34f58629c378450382251c29da84f37b/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /run/user/1000/app/com.jetbrains.PyCharm-Community/pip-wheel-6c73n6uu
cwd: /run/user/1000/app/com.jetbrains.PyCharm-Community/pip-install-wvz_dad4/postal_34f58629c378450382251c29da84f37b/
Complete output (39 lines):
/home/mike/PycharmProjects/govpole_prototype/venv/lib/python3.9/site-packages/setuptools/dist.py:691: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
/home/mike/PycharmProjects/govpole_prototype/venv/lib/python3.9/site-packages/setuptools/dist.py:691: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/postal
copying postal/expand.py -> build/lib.linux-x86_64-3.9/postal
copying postal/tokenize.py -> build/lib.linux-x86_64-3.9/postal
copying postal/near_dupe.py -> build/lib.linux-x86_64-3.9/postal
copying postal/parser.py -> build/lib.linux-x86_64-3.9/postal
copying postal/init.py -> build/lib.linux-x86_64-3.9/postal
copying postal/normalize.py -> build/lib.linux-x86_64-3.9/postal
copying postal/dedupe.py -> build/lib.linux-x86_64-3.9/postal
copying postal/token_types.py -> build/lib.linux-x86_64-3.9/postal
creating build/lib.linux-x86_64-3.9/postal/utils
copying postal/utils/enum.py -> build/lib.linux-x86_64-3.9/postal/utils
copying postal/utils/omitted.py -> build/lib.linux-x86_64-3.9/postal/utils
copying postal/utils/init.py -> build/lib.linux-x86_64-3.9/postal/utils
copying postal/utils/encoding.py -> build/lib.linux-x86_64-3.9/postal/utils
creating build/lib.linux-x86_64-3.9/postal/tests
copying postal/tests/_test_near_dupes.py -> build/lib.linux-x86_64-3.9/postal/tests
copying postal/tests/test_expand.py -> build/lib.linux-x86_64-3.9/postal/tests
copying postal/tests/init.py -> build/lib.linux-x86_64-3.9/postal/tests
copying postal/tests/test_parser.py -> build/lib.linux-x86_64-3.9/postal/tests
copying postal/pyutils.h -> build/lib.linux-x86_64-3.9/postal
running build_ext
building 'postal._expand' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/postal
x86_64-unknown-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -fPIC -I/usr/local/include -I/home/mike/PycharmProjects/govpole_prototype/venv/include -I/usr/include/python3.9 -c postal/pyexpand.c -o build/temp.linux-x86_64-3.9/postal/pyexpand.o -std=c99
postal/pyexpand.c:2:10: fatal error: libpostal/libpostal.h: No such file or directory
2 | #include <libpostal/libpostal.h>
| ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-unknown-linux-gnu-gcc' failed with exit code 1

ERROR: Failed building wheel for postal
Running setup.py clean for postal
Failed to build postal
Installing collected packages: postal
Running setup.py install for postal ... error
ERROR: Command errored out with exit status 1:
command: /home/mike/PycharmProjects/govpole_prototype/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/run/user/1000/app/com.jetbrains.PyCharm-Community/pip-install-wvz_dad4/postal_34f58629c378450382251c29da84f37b/setup.py'"'"'; file='"'"'/run/user/1000/app/com.jetbrains.PyCharm-Community/pip-install-wvz_dad4/postal_34f58629c378450382251c29da84f37b/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /run/user/1000/app/com.jetbrains.PyCharm-Community/pip-record-9fgrsfdg/install-record.txt --single-version-externally-managed --compile --install-headers /home/mike/PycharmProjects/govpole_prototype/venv/include/site/python3.9/postal
cwd: /run/user/1000/app/com.jetbrains.PyCharm-Community/pip-install-wvz_dad4/postal_34f58629c378450382251c29da84f37b/
Complete output (39 lines):
/home/mike/PycharmProjects/govpole_prototype/venv/lib/python3.9/site-packages/setuptools/dist.py:691: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
/home/mike/PycharmProjects/govpole_prototype/venv/lib/python3.9/site-packages/setuptools/dist.py:691: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/postal
copying postal/expand.py -> build/lib.linux-x86_64-3.9/postal
copying postal/tokenize.py -> build/lib.linux-x86_64-3.9/postal
copying postal/near_dupe.py -> build/lib.linux-x86_64-3.9/postal
copying postal/parser.py -> build/lib.linux-x86_64-3.9/postal
copying postal/init.py -> build/lib.linux-x86_64-3.9/postal
copying postal/normalize.py -> build/lib.linux-x86_64-3.9/postal
copying postal/dedupe.py -> build/lib.linux-x86_64-3.9/postal
copying postal/token_types.py -> build/lib.linux-x86_64-3.9/postal
creating build/lib.linux-x86_64-3.9/postal/utils
copying postal/utils/enum.py -> build/lib.linux-x86_64-3.9/postal/utils
copying postal/utils/omitted.py -> build/lib.linux-x86_64-3.9/postal/utils
copying postal/utils/init.py -> build/lib.linux-x86_64-3.9/postal/utils
copying postal/utils/encoding.py -> build/lib.linux-x86_64-3.9/postal/utils
creating build/lib.linux-x86_64-3.9/postal/tests
copying postal/tests/_test_near_dupes.py -> build/lib.linux-x86_64-3.9/postal/tests
copying postal/tests/test_expand.py -> build/lib.linux-x86_64-3.9/postal/tests
copying postal/tests/init.py -> build/lib.linux-x86_64-3.9/postal/tests
copying postal/tests/test_parser.py -> build/lib.linux-x86_64-3.9/postal/tests
copying postal/pyutils.h -> build/lib.linux-x86_64-3.9/postal
running build_ext
building 'postal._expand' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/postal
x86_64-unknown-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -fPIC -I/usr/local/include -I/home/mike/PycharmProjects/govpole_prototype/venv/include -I/usr/include/python3.9 -c postal/pyexpand.c -o build/temp.linux-x86_64-3.9/postal/pyexpand.o -std=c99
postal/pyexpand.c:2:10: fatal error: libpostal/libpostal.h: No such file or directory
2 | #include <libpostal/libpostal.h>
| ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-unknown-linux-gnu-gcc' failed with exit code 1
----------------------------------------
ERROR: Command errored out with exit status 1: /home/mike/PycharmProjects/govpole_prototype/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/run/user/1000/app/com.jetbrains.PyCharm-Community/pip-install-wvz_dad4/postal_34f58629c378450382251c29da84f37b/setup.py'"'"'; file='"'"'/run/user/1000/app/com.jetbrains.PyCharm-Community/pip-install-wvz_dad4/postal_34f58629c378450382251c29da84f37b/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /run/user/1000/app/com.jetbrains.PyCharm-Community/pip-record-9fgrsfdg/install-record.txt --single-version-externally-managed --compile --install-headers /home/mike/PycharmProjects/govpole_prototype/venv/include/site/python3.9/postal Check the logs for full command output.
`

[mzpqnxow]

When setuptools runs, it's not finding the headers (and probably the shared library also)

This can be fixed with the following, before running pip or python setup.py build:

$ export CFLAGS='-I/path/to/libpost/include -L/path/to/libpostal/lib`
# pip install, or python setup.py if you have it locally

If that doesn't help, then read the below. It will build both from git master

Forgive any typos, hopefully you get the idea- the point is that you need a few things:

• setuptools to know where libpostal is when you try to build pypostal. This includes both the headers (include files) and the shared library (.so files)
  • Set this via CFLAGS, when you buld pypostal
• The runtime linker/loader to know where the shared library is
  • Set this via LD_LIBRARY_PATH when you run python and try to import it. You can avoid this by statically linking libpostal into pypostal but I'm leaving that out to avoid confusing things. I didn't check to see how friendly the Makefile / configure script was to this, but it can often be as easy as adding -static to the CFLAGS when you build the python package

There are cases where neither of these explicit actions are necessary, but it depends on your distribution and where you choose to install the libraries and headers. I prefer to isolate third-party software to its own directory in /opt rather than dumping everything in /usr/local . The benefit of /usr/local, is a lot of build systems know to look there without being told. The benefit of using /opt/<package>-<version> is you can have multiple versions of packages and manage them at runtime using LD_LIBRARY_PATH and CFLAGS as needed. You probably don't need that here though :)

Feel free to change any of the variables below, just make sure they're consistent. Note that /etc/ld.so.conf & friends won't expand the environment variable, you have to put the full expanded path in, so if you change PREFIX, make sure anything you put in ld.so.conf matches up

    $ DATADIR=/home/mike/PycharmProjects/libpostal_data  <-- I don't necessarily recommend putting it here, see pycharm note below
    $ PREFIX=/opt/libpostal
    $ BUILDDIR=/home/mike/build
    $ mkdir -p $BUILDDIR && pushd ~/build
    $ git clone https://github.com/openvenues/pypostal
    $ git clone https://github.com/openvenues/libpostal
    $ pushd libpostal
    $ ./bootstrap.sh
    $ ./configure --datadir="$DATADIR" --prefix="$PREFIX"
    $ make -j4
    $ sudo make install
    $ sudo /usr/bin/libtool --mode=finish "$PREFIX/lib"
    $ popd
    $ pushd pypostal
    # See if it will build now ...
    $ CFLAGS="-I$PREFIX/include -L$PREFIX/lib" python setup.py build
    $ python setup.py install

If you have an error upon from postal.parser import parse_address, you will need to set your $LD_LIBRARY_PATH:

$ LD_LIBRARY_PATH=$PREFIX/lib:$LD_LIBRARY_PATH python test.py

If the LD_LIBRARY_PATH step was necessary, you can make it persistent by adding the following to either /etc/ld.so.conf:

# Assumes PREFIX=/opt/libpostal
/opt/libpostal/lib

Optionally, if your distribution supports this structure (Debian-based distributions do) you can create a file /etc/ld.so.conf.d/libpostal.conf with the same contents- it will be sourced by /etc/ld.so.conf when you run sudo ldconfig -a

WARNING: Putting the data directory there may cause PyCharm to index the gigantic .dat files, depending on your configuration- this will be very unpleasant