ENH: Use a neutral format to have lossless interface with scipp, Astropy, Xarray

[loco-philippe]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

Each tool has a specific structure for processing multidimensional data with the following consequences:

• interfaces dedicated to each tool,
• partially processed data,
• no unified representation of data structures

Feature Description

The proposed format (see jupyter notebook, github repository, PyPI package ) is based on the following principles:

• neutral format available for tabular or multidimensional tools (e.g. Numpy, pandas, xarray, scipp, astropy),
• taking into account a wide variety of data types as defined in NTV format,
• high interoperability: reversible (lossless round-trip) interface with tabular or multidimensional tools,
• reversible and compact JSON format,
• Ease of sharing and exchanging multidimensional and tabular data,

Alternative Solutions

A first tool exists (NTV-pandas) but only deals with the JSON interface and the analysis of tabular structures. It would therefore be extended to multidimensional structures.

Additional Context

numpy/numpy#12481 (comment)
astropy/astropy#16286
pydata/xarray#8927
scipp/scipp#3422