Use implicit row constructors. Optionally skip nested rows. Nullability bug fixes.

[ryannedolan]

Summary

TableResolver and DataType.Struct can now optionally skip nested Rows. This is useful for pipelines that can't handle complex record types.

ScriptImplementor will now replace explicit Row(...) constructors with implicit (...) row constructors, which is better supported by Flink before 1.18.

Added support for Avro arrays.

Small bug fixes related to nullability of Avro fields.

Details

Added dropNestedRows method to DataType.Struct. Added mapStruct method to TableResolver to enable applying this and similar data type transformations to rows. Adapters can use this as an easy way to deal with complex schemas.

Flink SQL prior to 1.18 doesn't support arbitrary expressions inside an explicit row constructor, so we now rewrite these using an implicit row constructor. This works when the number of elements in the row is more than one.

Added support for Avro arrays, and added a commented-out impl of Avro maps. Maps aren't un-parseable in this version of Calcite, so Avro maps will need to wait for a version bump.

Testing

New unit tests covering:

• avro --> rel and rel --> type conversions with nested rows
• nullability of nested avro rows
• drop nested rows

Also, spot-tested in a Kafka-->Kafka pipeline.