No_null_schema inference option in pl.DataFrame

[clayop]

Description

I am collecting some dirty data. While I convert json-ed data into pl.DataFrame, I met the following error.

could not append value: "21.04.22" of type: str to the builder; make sure that all rows have the same schema or consider increasing infer_schema_length

This is raised because the data I requested does not have any(null) in infer_schema_length rows, and Polars set this column as null, but 21.04.22 is shown after the rows. I resolved it by increase the length, but this consequently caused another inference error in another column.

The core problem is null type column. The safer way to deal with dirty data is to regard them as string. So if sort of no_null_schema exists, it would be really helpful to deal with dirty data.