[pyarrow] Missing optional dependency 'pyarrow.parquet' after updating pyarrow to v14.0.1

[gbarrospfrmwk]

Issue: After updating pyarrow from v12.0.1 to v14.0.1 the import error is thrown when I try to use the read_parquet function

Error ImportError: Missing optional dependency 'pyarrow.parquet'. pyarrow is required for Parquet support. "

Env: AWS Python Lambda 3.10

This error occurs only in the AWS Lambda environment, locally the project runs fine, and I already updated every possible library, checking the version with poetry show -o

pyproject.toml

[tool.poetry.dependencies]
python = "^3.10"
boto3 = "^1.29.1"
geopandas = "^0.14.1"
pyarrow = "^14.0.1"
psycopg2-binary = "^2.9.9"
sqlalchemy = "^2.0.23"
requests = "^2.31.0"
tox = "^4.11.3"
tenacity = "^8.2.3"
memoization = "^0.4.0"
pydantic = "^2.5.1"
urllib3 = "1.26.18"
certifi = "2023.7.22"
tomlkit = "^0.12.3"
virtualenv = "^20.24.6"
pandas = "^2.1.3"

[m-richards]

There's not really a lot we can do about this on the geopandas side. This seems to be an environment/ installation issue related to your lambda environment. Possible steps for debugging would be to log the output of pip list / conda list inside the lambda (your poetry file doesn't actually tell what actually was installed) and checking if you can import pyarrow directly without issue.

[gbarrospfrmwk]

After trying to import it directly I got an error about glibc version, so that's really about my lambda environment. I found out that AWS will launch Python 3.12 with glibc 2.34 next month, the current python runtime uses Amazon Linux 2 which has glibc 2.26, but the lib expects glibc 2.27. It would be nice to improve the geopandas import error, so we can identify the root cause.