You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The flytekit remote client relies on gRPC for communication with the control plane (FlyteAdmin). However, our experience has revealed numerous issues within Python gRPC, leading to test failures. Additionally, users frequently encounter challenges when working with specific versions of gRPC.
In order to eliminate the gRPC Python dependency from flytekit, our goal is to develop a Rust crate to serve as a replacement for the flytekit remote client. To achieve this, we plan to utilize Pyo3 to generate a Python binding, allowing flytekit to seamlessly integrate and utilize the new implementation.
Goal: What should the final outcome look like, ideally?
flyteorg/flytekit#2377 is a POC of a rust remote client that allows you to get/create tasks without Python grpc
Add a flag (enable_rust) to the FlyteRemote class. When this flag is true, FlyteRemote will use the rust implementation.
To address the issues with Python gRPC in the Flytekit remote client, a Rust-based replacement is proposed. This approach aims to eliminate Python gRPC dependencies by using Pyo3 for Python bindings, ensuring seamless integration with Flytekit. A proof of concept is available in this pull request, demonstrating task operations without Python gRPC. The proposed solution includes an enable_rust flag in FlyteRemote to switch to the Rust implementation. Future tasks involve testing the Rust client, ensuring output consistency between Rust and Python implementations, and integrating OAuth and credential passing.
Motivation: Why do you think this is important?
The flytekit remote client relies on gRPC for communication with the control plane (FlyteAdmin). However, our experience has revealed numerous issues within Python gRPC, leading to test failures. Additionally, users frequently encounter challenges when working with specific versions of gRPC.
In order to eliminate the gRPC Python dependency from flytekit, our goal is to develop a Rust crate to serve as a replacement for the flytekit remote client. To achieve this, we plan to utilize Pyo3 to generate a Python binding, allowing flytekit to seamlessly integrate and utilize the new implementation.
Goal: What should the final outcome look like, ideally?
flyteorg/flytekit#2377 is a POC of a rust remote client that allows you to get/create tasks without Python grpc
Add a flag (enable_rust) to the FlyteRemote class. When this flag is true, FlyteRemote will use the rust implementation.
Follow up Issues
Describe alternatives you've considered
Use existing flytekit remote client
Propose: Link/Inline OR Additional context
grpcio>=1.59.0
related to grpc thread pool hanging ingrpc_google_default_credentials_create
grpc/grpc#36265Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?
The text was updated successfully, but these errors were encountered: