Type Coercion In Pandas

[OmegaDroid]

The type coercion in pandas is currently inefficient due to recording the error on the row forcing the data type to be object rather than a number type.

If we drop the specific error message w can keep the value type as number and find new NaN values to report as being invalid.

We would need to check for null values in 2 stages.

1. Perform all null transformations, record all null values in the row.
2. Perform all value transformations, recording all new null values, dropping these new rows and reporting Failed to coerce 'x' (<value>): Failed to convert <value> to <type>: Row ...

This should keep the correct data types on the dataframe improving transformation efficiency.