Raise error on invalid input shape in Map

[maxnoe]

Description

Connected to #5179 I wanted to see what fails if instead of silently converting the input data into the expected shape, an error is raised.

It seems most cases where this fails is just tests "being lazy" with toy data not in the correct shape and relying on this reshaping, not any actual data files being loaded that require this treatment.

[AtreyeeS]

Thanks @maxnoe ! I agree it is rather strange to silently re-order the data (and a relief it does not affect any major results!) I think it might still be useful to allow for broadcastable shapes instead of the hard equality you introduced.

[adonath]

Thanks @maxnoe, I agree this is the safer behavior! I'm not sure when this was introduced, but it must have been there for many years already. This tells us at least that it did not lead to frequent problems and people possibly relied on it without issues (we don't know...). So I am rather inclined to formally deprecate this behavior instead. On the other hand it seems the behavior is not documented either...

[AtreyeeS]

@adonath does this need to be deprecated? Adding quality checks on input data is more of a new feature/clean up, no?

[adonath]

@AtreyeeS I don't have a strong opinion here, but the fact that this went without complaint for such a long time might mean that some users rely on the behavior (as I said...we don't really know). In any case it will be only a very tiny fraction, so we can change without deprecation.

[maxnoe]

I allowed broadcasting now, but the tests still fail on shapes that we might think of as compatible but are not to the numpy broadcasting rules:

FAILED gammapy/maps/hpx/tests/test_ndmap.py::test_partial_hpx_map_cutout - ValueError: Input shape (1, 1, 25) is not compatible with shape from geometry (1, 25)
FAILED gammapy/maps/hpx/tests/test_ndmap.py::test_hpx_map_to_region_nd_map - ValueError: Input shape (10,) is not compatible with shape from geometry (10, 1, 1)
FAILED gammapy/maps/hpx/tests/test_ndmap.py::test_hpxmap_read_healpy - ValueError: Input shape (12288, 1024) is not compatible with shape from geometry (12582912,)
FAILED gammapy/maps/region/tests/test_ndmap.py::test_region_nd_map_plot_two_axes - ValueError: Input shape (6,) is not compatible with shape from geometry (3, 2, 1, 1)
FAILED gammapy/maps/region/tests/test_ndmap.py::test_label_axis_io - ValueError: Input shape (10,) is not compatible with shape from geometry (2, 5, 1, 1)
FAILED gammapy/maps/region/tests/test_ndmap.py::test_region_nd_io_ogip - ValueError: Input shape (12,) is not compatible with shape from geometry (12, 1, 1)
FAILED gammapy/maps/region/tests/test_ndmap.py::test_region_nd_io_ogip_arf - ValueError: Input shape (12,) is not compatible with shape from geometry (12, 1, 1)
FAILED gammapy/maps/region/tests/test_ndmap.py::test_region_nd_io_gadf - ValueError: Input shape (2,) is not compatible with shape from geometry (2, 1, 1)
FAILED gammapy/maps/region/tests/test_ndmap.py::test_region_nd_io_gadf_no_region - ValueError: Input shape (2,) is not compatible with shape from geometry (2, 1, 1)
FAILED gammapy/maps/region/tests/test_ndmap.py::test_region_nd_io_gadf_rad_axis - ValueError: Input shape (6,) is not compatible with shape from geometry (3, 2, 1, 1)
FAILED gammapy/maps/region/tests/test_ndmap.py::test_region_nd_map_interp_no_region - ValueError: Input shape (2, 3) is not compatible with shape from geometry (3, 2, 1, 1)
FAILED gammapy/maps/wcs/tests/test_ndmap.py::test_get_spectrum - ValueError: Input shape (3,) is not compatible with shape from geometry (3, 1, 1)
FAILED gammapy/maps/wcs/tests/test_ndmap.py::test_get_spectrum_type - ValueError: Input shape (3,) is not compatible with shape from geometry (3, 1, 1)
FAILED gammapy/maps/wcs/tests/test_ndmap.py::test_get_spectrum_weights - ValueError: Input shape (3,) is not compatible with shape from geometry (3, 1, 1)
FAILED gammapy/maps/wcs/tests/test_ndmap.py::test_to_region_nd_map_histogram_basic - ValueError: Input shape (3, 100) is not compatible with shape from geometry (3, 100, 1, 1)
FAILED gammapy/maps/wcs/tests/test_ndmap.py::test_to_region_nd_map_histogram_advanced - ValueError: Input shape (2, 3, 4) is not compatible with shape from geometry (2, 3, 4, 1, 1)

as numpy only allows broadcasting on the left side, the last dimensions have to match.

I.e. (3, 100) is not broadcastable to (3, 100, 1, 1), but would be to (1, 1, 3, 100).

[maxnoe]

Ah, btw.: I should mention that these are only the tests failing in gammapy.maps, running the whole test suite, many, many, many more tests are failing. I'll compile a list of shape mismatches to see if all these are benign (not actually changing data order).

[maxnoe]

These are the unique shape combinations that make the tests fail:

(1000,), (1000, 1, 1)
(100050,), (2001, 50, 1, 1)
(100,), (100, 1, 1)
(100, 3, 3), (100, 1, 3, 3)
(10, 10, 10), (10, 1, 10, 10)
(10,), (10, 1, 1)
(10, 1), (10, 1, 1)
(10, 1, 1), (10, 1, 1, 1)
(10,), (2, 5, 1, 1)
(10, 25, 17), (10, 1, 25, 17)
(108, 1, 1), (108, 1, 1, 1)
(109, 1, 1), (109, 1, 1, 1)
(11098,), (62, 179, 1, 1)
(11,), (11, 1, 1)
(1, 1, 128, 128), (1, 128, 128)
(1, 1, 25), (1, 25)
(1, 1, 260, 260), (1, 260, 260)
(11, 300), (11, 300, 1, 1)
(1, 1, 32, 40), (1, 32, 40)
(1, 1, 40, 50), (1, 40, 50)
(1, 1, 50, 50), (1, 50, 50)
(1, 1, 64, 64), (1, 64, 64)
(12, 1, 1), (12, 1, 1, 1)
(1, 2), (1, 2, 1, 1)
(12,), (12, 1, 1)
(12288, 1024), (12582912,)
(1, 3), (1, 3, 1, 1)
(13,), (13, 1, 1)
(132,), (12, 11, 1, 1)
(14,), (14, 1, 1, 1)
(14, 20, 20), (14, 1, 20, 20)
(15, 1, 1), (15, 1, 1, 1)
(16, 1, 1), (16, 1, 1, 1)
(167,), (167, 1, 1)
(17, 300), (17, 300, 1, 1)
(198,), (3, 66, 1, 1)
(200,), (200, 1, 1)
(20, 1, 1), (20, 1, 1, 1)
(2, 10), (2, 10, 1, 1)
(21,), (7, 3, 1, 1)
(2, 20, 20), (2, 1, 20, 20)
(2,), (2, 1, 1)
(2,), (2, 1, 1, 1)
(2, 2), (2, 2, 1, 1)
(2, 25, 25), (2, 1, 25, 25)
(2, 25, 50), (2, 1, 25, 50)
(2, 3), (3, 2, 1, 1)
(2, 3, 4), (2, 3, 4, 1, 1)
(24,), (24, 1, 1)
(2551,), (2551, 1, 1)
(2870,), (70, 41, 1, 1)
(2, 8, 8), (2, 1, 8, 8)
(30, 22, 22), (30, 1, 22, 22)
(30, 28, 28), (30, 1, 28, 28)
(30,), (30, 1, 1)
(3, 100), (3, 100, 1, 1)
(3, 1, 2), (3, 1, 1, 2)
(32, 3, 3), (32, 1, 3, 3)
(33, 10, 10), (33, 1, 10, 10)
(3,), (3, 1, 1)
(34, 1), (34, 1, 1, 1)
(3, 4, 4), (3, 1, 4, 4)
(3, 5, 10), (3, 1, 5, 10)
(3, 66), (3, 66, 1, 1)
(3, 8, 8), (3, 1, 8, 8)
(4, 1, 90), (4, 90)
(4, 25, 25), (4, 1, 25, 25)
(4,), (4, 1, 1)
(4, 5, 5), (4, 1, 5, 5)
(48,), (48, 1, 1, 1)
(500,), (5, 10, 10, 1, 1)
(50, 1, 1), (50, 1, 1, 1)
(5,), (5, 1, 1)
(6, 1, 5986), (6, 5986)
(6,), (3, 2, 1, 1)
(6,), (6, 1, 1)
(66,), (1, 66, 1, 1)
(70, 41), (70, 41, 1, 1)
(7,), (7, 1, 1)
(80,), (80, 1, 1)
(8,), (8, 1, 1)
(8,), (8, 1, 1, 1)
(9, 1, 1), (9, 1, 1, 1)

I don't see only a few that are not additional size 1 dimensions:

(100050,), (2001, 50, 1, 1)
(11098,), (62, 179, 1, 1)
(12288, 1024), (12582912,)
(132,), (12, 11, 1, 1)
(198,), (3, 66, 1, 1)
(2870,), (70, 41, 1, 1)
(500,), (5, 10, 10, 1, 1)

[registerrier]

Thanks @maxnoe . I think the reshape was added when RegionNDMap were introduced because of their peculiar shapes. That was probably not the best way to handle this.
Maybe a better solution is to treat the specific case of RegionNDMap with a proper data setter and not the base class version.