BUG: Uncertainty type ignored in Cubeviz spectral extraction

[pllim]

The current documentation (at the time of original posting this issue) at https://jdaviz.readthedocs.io/en/latest/cubeviz/plugins.html#spectral-extraction does not mention at all how uncertainty and mask arrays are used in the extraction process. The mask is yet another issue (#2714); here I am only focusing on uncertainty:

jdaviz/jdaviz/configs/cubeviz/plugins/spectral_extraction/spectral_extraction.py

Lines 207 to 222 in b563ed3

	uncert_cube = self._app._jdaviz_helper._loaded_uncert_cube
	# This plugin collapses over the *spatial axes* (optionally over a spatial subset,
	# defaults to ``No Subset``). Since the Cubeviz parser puts the fluxes
	# and uncertainties in different glue Data objects, we translate the spectral
	# cube and its uncertainties into separate NDDataArrays, then combine them:
	if self.aperture.selected != self.aperture.default_text:
	nddata = spectral_cube.get_subset_object(
	subset_id=self.aperture.selected, cls=NDDataArray
	)
	uncertainties = uncert_cube.get_subset_object(
	subset_id=self.aperture.selected, cls=StdDevUncertainty
	)
	else:
	nddata = spectral_cube.get_object(cls=NDDataArray)
	uncertainties = uncert_cube.get_object(cls=StdDevUncertainty)

This was originally implemented by @bmorris3 via #2039 . In the code above, uncertainty cube is assumed to be always be in standard deviation.

However, Cubeviz parser actually allows at least 3 different kinds of uncertainty (inverse variance, variance, standard deviation, and who knows what else). The uncertainty type was only to detect whether a cube falls in that category, we do not have any extra processing to make sure it is actually always standard deviation.

jdaviz/jdaviz/configs/cubeviz/plugins/parsers.py

Lines 19 to 21 in b563ed3

	EXT_TYPES = dict(flux=['flux', 'sci', 'data'],
	uncert=['ivar', 'err', 'var', 'uncert'],
	mask=['mask', 'dq', 'quality'])

As a result, if the uncertainty cube is not really standard deviation, it might be wrong to assume so, and that wrong assumption would propagate to it being handled improperly during extraction.

While, there appears to be some checks in https://github.com/glue-viz/glue-astronomy/blob/main/glue_astronomy/translators/nddata.py . I am not sure if those checks are actually used always due to the way we store uncertainty cube as a separate Data object, and not as data.uncertainty.

For instance, at one point, glue-astronomy translator looks for data.meta.get('uncertainty_type', 'std') but we never set that in Cubeviz parser. As a result, any of our code that goes through that logic would always return 'std' regardless.

The scope of this ticket is to confirm whether or not my concern of a bug is real. And if it is, what is necessary to fix. Thanks.

🐱

[bmorris3]

The glue-astronomy NDData translator will represent any uncertainties in glue Data entry for the uncertainty as StdDevUncertainty by default (see NDData source here) when you ask for data.to_object. But note that the relevant translator from input->Data depends on what the input data are. I believe Cubeviz spectral extraction returns NDDataArrays, but the jwst products from MAST will be parsed via the glue-astro translator for Spectrum1D objects (in this block).

Both the NDData and Spectrum1D translators check for a metadata keyword called uncertainty_type, and uses the associated value to determine which type of uncertainties are given (source here). I believe I implemented this approach knowing that we could set an uncertainty_type key in meta in the parser, if we knew how to identify the uncertainty type correctly.

For your ticket, the next step is to check if (1) any JWST cubes have non-stddev uncertainties, (2) if there's a keyword that specifies the uncertainty type in the JWST spectral cubes used in Cubeviz, (3) setting data.meta['uncertainty_type'] if you find (2) and it's not stddev.