Is there a way to get a DataFrameModel from existing pandas.DataFrame?

[Felix-neko]

Hi folks! And thank you for your project.

I want to use pandera to validate input dataframes of data transformation functions I have to write -- and I have a question.

Is it possible to generate a DataFrameModel from existing pandas.DataFrame?

Something like infer_schema(...), but infer_model(...)?

Is it possible?

[westurner]

• ENH,SCH: Function that maps from (dataframelib, column datatype) to (pandera, column datatype)?

• Pandas DataFrame column datatypes otoh:

  • python primitives
  • objects: object(), datetime.datetime
  • numpy dtypes like float64
    • https://numpy.org/doc/stable/reference/arrays.dtypes.html#attributes
  • numpy dtypes like ndarray and matrix
  • numpy ufuncs:
    https://numpy.org/doc/stable/user/basics.ufuncs.html#overriding-ufunc-behavior
  • units
    • pint
      https://pypi.org/project/pint (ufuncs iirc)
      • https://github.com/hgrecco/pint-pandas
        pandas.DataFrame(,dtype="pint[lbf ft]")
    • sympy.physics.units
      https://pypi.org/project/sympy
    • astropy.units
      https://pypi.org/project/astropy
    • unyt
      https://pypi.org/project/unyt
  • error propagation:
    • uncertainties:
      https://pypi.org/project/uncertainties/
    • sigfig:
      https://pypi.org/project/sigfig/
  • money types:
    • decimal.Decimal: https://docs.python.org/3/library/decimal.html
    • py-moneyed:
  • pandas types:
    • https://pandas.pydata.org/docs/user_guide/basics.html#basics-dtypes
    • https://pandas.pydata.org/docs/reference/arrays.html#pandas-arrays-scalars-and-data-types
    • https://arrow.apache.org/docs/python/generated/pyarrow.DataType.html#pyarrow.DataType.to_pandas_dtype
  • tensors,:
    • https://pytorch.org/docs/stable/tensors.html#data-types
    • https://pytorch.org/docs/stable/tensors.html#torch.Tensor
    • https://pytorch.org/torcharrow/beta/column.html#data-types
    • https://pytorch.org/torcharrow/beta/generated/torcharrow.DataFrame.to_tensor.html

[kykyi]

@Felix-neko I don't think this is possible atm, just to check, would this be to programatically ensure different dataframes follow the same schema, or would this be to get a dataframe, extract the schema, and save it to a file?