AnalysisException/DATATYPE_MISMATCH error when generating summary dataframe from sources that have a column named "summary"

[artruk]

Expected Behavior

Current Behavior

Generating summary dataframe using DataAnalyzer seems to fail whenever the source being analyzed has a column named "summary"

Steps to Reproduce (for bugs)

import dbldatagen as dg

df = spark.range(10).withColumnRenamed("id", "summary")
summary_df = dg.DataAnalyzer(sparkSession=spark, df = df).summarizeToDF()

Context

Your Environment

• dbldatagen version used:
• Databricks Runtime version:
• Cloud environment used:

[ronanstokes-db]

we'll add a fix to this in the next hotfix.

In the meantime you can rename the "summary" field to something else - but avoid using leading underscores as these may conflict with internal column names

[ronanstokes-db]

Fixed in hotfix as of 05/22/24