[SPARK-48322][SPARK-42965][SQL][CONNECT][PYTHON] Drop internal metadata in DataFrame.schema

[zhengruifeng]

What changes were proposed in this pull request?

Drop internal metadata in DataFrame.schema

Why are the changes needed?

Internal metadata might be leaked in both Spark Connect and Spark Classic,

e.g. in Spark Classic

In [9]: spark.range(10).select(sf.lit(1).alias("key"), "id").groupBy("key").agg(sf.max("id")).schema.json()
Out[9]: '{"fields":[{"metadata":{},"name":"key","nullable":false,"type":"integer"},{"metadata":{"__autoGeneratedAlias":"true"},"name":"max(id)","nullable":true,"type":"long"}],"type":"struct"}'

What make it worse is that internal metadata maybe leaked in different cases of Spark Connect and Spark Classic, so need to add additional _drop_meta in Pandas APIs to make assertions work.

Does this PR introduce any user-facing change?

yes, the internal metadata will be dropped in DataFrame.schema, e.g. the __autoGeneratedAlias in the example query

before:

In [9]: spark.range(10).select(sf.lit(1).alias("key"), "id").groupBy("key").agg(sf.max("id")).schema.json()
Out[9]: '{"fields":[{"metadata":{},"name":"key","nullable":false,"type":"integer"},{"metadata":{"__autoGeneratedAlias":"true"},"name":"max(id)","nullable":true,"type":"long"}],"type":"struct"}'

after:

In [2]: spark.range(10).select(sf.lit(1).alias("key"), "id").groupBy("key").agg(sf.max("id")).schema.json()
Out[2]: '{"fields":[{"metadata":{},"name":"key","nullable":false,"type":"integer"},{"metadata":{},"name":"max(id)","nullable":true,"type":"long"}],"type":"struct"}

How was this patch tested?

CI

Was this patch authored or co-authored using generative AI tooling?

No

[zhengruifeng]

cc @cloud-fan and @HyukjinKwon

[zhengruifeng]

cc @cloud-fan and @HyukjinKwon would you mind taking a look? thanks

[cloud-fan]

thanks, merging to master!

[cloud-fan]

can we update the Does this PR introduce any user-facing change? section? This is a user-visible change, as getting StructField metadata is a public API.

[zhengruifeng]

can we update the Does this PR introduce any user-facing change? section? This is a user-visible change, as getting StructField metadata is a public API.

done, thanks

[HyukjinKwon]

Merged to master.

[zhengruifeng]

I am going to revert this PR, since it causes some streaming integration tests fail

will try to fix this issue again

[gatorsmile]

@zhengruifeng for such a breaking change, we should send an email to the dev list before merging the PR.

[zhengruifeng]

got it, it has been reverted