Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use latest compatible grpcio-tools version and unlock support for Python 3.11 #57

Open
discordianfish opened this issue Feb 6, 2023 · 1 comment

Comments

@discordianfish
Copy link
Member

Currently blocked to 1.48.2 because of TF 2.x limitation of protobuf < 3.20 (and TF is currently used in Ray RL Lib). This also limit python version to 3.11 as it requires a newer grpcio version (above 1.48)

Files affected:

  • Engine Docker File
  • diambra-engine python package build-pip-package
  • GH actions limited to python 3.10

Other relevant info

Precondition on protobuf changed between:
grpcio-tools 1.48.2 https://github.com/grpc/grpc/blob/5e97a6003d0e44bdf3c4234d37b9509d02bf55ea/tools/distrib/python/grpcio_tools/setup.py#L293
and 1.49: https://github.com/grpc/grpc/blob/8f8edfd04b46ee67f90454b3f6a70aa58ff82c2d/tools/distrib/python/grpcio_tools/setup.py#L293

Compatibility problem of protobuf
https://stackoverflow.com/questions/71759248/importerror-cannot-import-name-builder-from-google-protobuf-internal

@discordianfish
Copy link
Member Author

To sum this up:

  • ray-rllib needs tensorflow 2.x
  • tensorflow 2.x needs protobuf (<3.20,>=3.9.2)
$ for v in 2.11.0 2.10.1 2.10.0 2.9.3 2.9.2 2.9.1 2.8.4; do echo "$v: $(curl -s https://pypi.org/pypi/tensorflow/$v/json | jq .info.requires_dist|grep protobuf)"; done
2.11.0:   "protobuf (<3.20,>=3.9.2)",
2.10.1:   "protobuf (<3.20,>=3.9.2)",
2.10.0:   "protobuf (<3.20,>=3.9.2)",
2.9.3:   "protobuf (<3.20,>=3.9.2)",
2.9.2:   "protobuf (<3.20,>=3.9.2)",
2.9.1:   "protobuf (<3.20,>=3.9.2)",
2.8.4:   "protobuf (<3.20,>=3.9.2)",
  • Since there can be only one version installed, we need to depend on protobuf (<3.20,>=3.9.2) as well
  • grpcio-tools generates the diambra-arena-engine code assuming a specific protobuf version
  • grpcio-tools depends on that protobuf version:
$ for v in 1.51.1 1.50.0 1.49.1 1.48.2; do echo "$v: $(curl -s https://pypi.org/pypi/grpcio-tools/$v/json | jq .info.requires_dist|grep protobuf)"; done
1.51.1:   "protobuf (<5.0dev,>=4.21.6)",
1.50.0:   "protobuf (<5.0dev,>=4.21.6)",
1.49.1:   "protobuf (<5.0dev,>=4.21.3)",
1.48.2:   "protobuf (<4.0dev,>=3.12.0)",

Therefor we need to:

  1. Generate diambra-arena-engine using a grpcio-tools version that generate code for a version matching protobuf (<3.20,>=3.9.2), e.g 1.48.2
  2. Use that version's dependency (protobuf (<4.0dev,>=3.12.0) in our setup.py

That will allow users to install ray-rllib where pip dep resolver should install protobuf <3.20,>=3.9.2 while allowing users not depending on ray/tf2 to also install <=3.12.0 and >=3.20.

@alexpalms alexpalms removed their assignment Mar 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants