[SPARK-52897][PYTHON] Update pandas to 2.3.1

[bjornjorgensen]

What changes were proposed in this pull request?

Upgrading from pandas==2.3.0 to pandas==2.3.1

Why are the changes needed?

Release Github

Release notes

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA

Was this patch authored or co-authored using generative AI tooling?

No.

[bjornjorgensen]

@zhengruifeng @yaooqinn @HyukjinKwon @dongjoon-hyun

[HyukjinKwon]

can we also fix https://github.com/apache/spark/blob/master/python/pyspark/pandas/supported_api_gen.py#L41?

[bjornjorgensen]

@HyukjinKwon are there another to that we need to update

I get this now

*****************************
* Building Python API docs. *
*****************************
Running Sphinx v4.5.0
/__w/spark/spark/python/pyspark/pandas/__init__.py:43: UserWarning: 'PYARROW_IGNORE_TIMEZONE' environment variable was not set. It is required to set this environment variable to '1' in both driver and executor sides if you use pyarrow>=2.0.0. pandas-on-Spark will set it for you but it does not work if there is a Spark context already launched.
  warnings.warn(
/__w/spark/spark/python/pyspark/pandas/supported_api_gen.py:117: UserWarning: Warning: pandas 2.3.1 is required; your version is 2.3.0
  warnings.warn(msg, UserWarning)

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/sphinx/config.py", line 332, in eval_config_file
    exec(code, namespace)
  File "/__w/spark/spark/python/docs/source/conf.py", line 33, in <module>
    generate_supported_api(output_rst_file_path)
  File "/__w/spark/spark/python/pyspark/pandas/supported_api_gen.py", line 102, in generate_supported_api
    _check_pandas_version()
  File "/__w/spark/spark/python/pyspark/pandas/supported_api_gen.py", line 118, in _check_pandas_version
    raise ImportError(msg)
ImportError: Warning: pandas 2.3.1 is required; your version is 2.3.0

[zhengruifeng]

this file is only used for old branches, let's skip it

[bjornjorgensen]

ok, I change all pandas==2.3.1 to pandas==2.3.0 in this file.

[bjornjorgensen]

@zhengruifeng I updated pandas version in the dev/spark-test-image/docs/Dockerfile and now all tests passes.

[the-sakthi]

LGTM!

[yaooqinn]

Merged to master, thank you @bjornjorgensen and all