Build instructions for mstch_py3 generated code

[tscholak]

Hi,
After installing thrift-0.0.1-cp38-cp38-linux_x86_64.whl, I'm now looking for a way to compile the code generated by thrift1 --gen mstch_py3 x.thrift. I'm getting a folder, gen-py3/x, with the following contents:

builders.pxd
builders.pyi
builders.pyx
clients.pxd
clients.pyi
clients.pyx
clients_wrapper.cpp
clients_wrapper.h
clients_wrapper.pxd
metadata.cpp
metadata.h
metadata.pxd
metadata.pyi
metadata.pyx
services.pxd
services.pyi
services.pyx
services_reflection.pxd
services_reflection.pyx
services_wrapper.cpp
services_wrapper.h
services_wrapper.pxd
types.h
types.pxd
types.pyi
types.pyx
types_fields.pxd
types_fields.pyx
types_reflection.pxd
types_reflection.pyx

I've been trying to understand how the tests are compiled, for instance those deposited in fbthrift/_build/thrift/lib/py3/test/gen-py3/binary, but without much luck. I am not familiar with Cmake.
When I try to use cython to compile each of the pyx files to cpp manually in this folder,

$ cython --fast-fail -3 --cplus -I./fbthrift/_build/thrift/lib/py3/cybld/ gen-py3/x/types.pyx -o gen-py3/x/types.cpp

, I am getting a lot of failed import errors. It even seems that there are cyclic dependencies between types and types_fields. Is there a way to generate a Cmake file with thrift1 as well? I'm really lost here.
I'd be glad for any advice. Thanks!

[tscholak]

update: I added a __init__.py, and that allowed me to generate cpp code from clients.pyx.
(see also https://github.com/facebook/fbthrift/blob/master/ThriftLibrary.cmake#L255)

[tscholak]

Ok, small progress. I generated gen-cpp2 and added a setup.py file in gen-py3:

from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules=cythonize(["x/types.pyx", "x/clients.pyx", "x/services.pyx"], language_level=3),
    include_dirs=["../."]
)

with this, I can compile three modules,

CPPFLAGS="--std=c++17" python setup.py build_ext --inplace

however, when I try to import the clients module, I'm getting an ImportError:

$ python
>>> import x.clients
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: [...]/gen-py3/x/clients.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZTIN5folly7futures6detail8CoreBaseE

and when I try to import the types module, I'm getting another one:

>>> import x.types
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: [...]/gen-py3/x/types.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3fLI46FLAGS_thrift_cpp2_protocol_reader_string_limitE

[tscholak]

OK, I overcame some of the missing symbol errors by recompiling the pyx files -lfolly -lthrift-core -lthriftcpp2. Now, the remaining symbols all seem to point to gen-cpp2. There is no makefile in there, I wonder how to build that...

[tscholak]

After a couple of days of trial and error, I figured out a setup.py that works for me. I post it here so that others don't have to go through what I had to overcome:

from setuptools import Extension, setup
from Cython.Build import cythonize

libraries = [
    "folly",
    "thriftcpp2",
]
extra_libraries = [
    "x.cpython-38-x86_64-linux-gnu",
]
extra_library_dirs = [
    "build/lib.linux-x86_64-3.8/x",
]
extensions = [
    Extension(
        "x.libx",
        [
            "gen-cpp2/x.cpp",
            "gen-cpp2/xAsyncClient.cpp",
            "gen-cpp2/x_processmap_binary.cpp",
            "gen-cpp2/x_processmap_compact.cpp",
            "gen-cpp2/x_constants.cpp",
            "gen-cpp2/x_data.cpp",
            "gen-cpp2/x_metadata.cpp",
            "gen-cpp2/x_types.cpp"
        ],
        include_dirs=[],
        libraries=libraries,
        library_dirs=[],
        extra_compile_args = ["--std=c++17"]
    ),
    Extension(
        "x.types",
        [
            "x/types.pyx",
            "fbthrift/thrift/lib/py3/metadata.cpp",
            "fbthrift/thrift/lib/py3/enums.cpp",
        ],
        include_dirs=["../."],
        libraries=libraries + extra_libraries,
        library_dirs=extra_library_dirs,
        extra_compile_args = ["--std=c++17"]
    ),
    Extension(
        "x.metadata",
        [
            "x/metadata.pyx",
            "fbthrift/thrift/lib/py3/metadata.cpp",
            "fbthrift/thrift/lib/py3/enums.cpp",
        ],
        include_dirs=["../."],
        libraries=libraries + extra_libraries,
        library_dirs=extra_library_dirs,
        extra_compile_args = ["--std=c++17"]
    ),
    Extension(
        "x.types_fields",
        [
            "x/types_fields.pyx",
            "fbthrift/thrift/lib/py3/metadata.cpp",
            "fbthrift/thrift/lib/py3/enums.cpp",
        ],
        include_dirs=["../."],
        libraries=libraries + extra_libraries,
        library_dirs=extra_library_dirs,
        extra_compile_args = ["--std=c++17"]
    ),
    Extension(
        "x.types_reflection",
        [
            "x/types_reflection.pyx",
            "fbthrift/thrift/lib/py3/metadata.cpp",
            "fbthrift/thrift/lib/py3/enums.cpp",
        ],
        include_dirs=["../."],
        libraries=libraries + extra_libraries,
        library_dirs=extra_library_dirs,
        extra_compile_args = ["--std=c++17"]
    ),
    Extension(
        "x.clients",
        [
            "x/clients.pyx",
            "x/clients_wrapper.cpp",
            "fbthrift/thrift/lib/py3/metadata.cpp",
            "fbthrift/thrift/lib/py3/enums.cpp",
        ],
        include_dirs=["../."],
        libraries=libraries + extra_libraries,
        library_dirs=extra_library_dirs,
        extra_compile_args = ["--std=c++17"]
    ),
    Extension(
        "x.services",
        [
            "x/services.pyx",
            "x/services_wrapper.cpp",
            "fbthrift/thrift/lib/py3/metadata.cpp",
            "fbthrift/thrift/lib/py3/enums.cpp",
        ],
        include_dirs=["../."],
        libraries=libraries + extra_libraries,
        library_dirs=extra_library_dirs,
        extra_compile_args = ["--std=c++17"]
    ),
    Extension(
        "x.services_reflection",
        [
            "x/services_reflection.pyx",
            "fbthrift/thrift/lib/py3/metadata.cpp",
            "fbthrift/thrift/lib/py3/enums.cpp",
        ],
        include_dirs=["../."],
        libraries=libraries + extra_libraries,
        library_dirs=extra_library_dirs,
        extra_compile_args = ["--std=c++17"]
    )
]

setup(
    ext_modules=cythonize(extensions, language_level=3)
)

It would be amazing if the thrift compiler would generate build scripts as well.