create_export_csv rounding should be sigfigs, not just digits

[dsweber2]

Specifically, this line:

covidcast-indicators/_delphi_utils_python/delphi_utils/export.py

Line 128 in a99327b

export_df = export_df.round({"val": 7, "se": 7})

Not currently a problem as far as I can tell, but if we ever get a signal whose data is of the size 1.423e-8, then this will round to zero.

I hacked together a function that does this correctly from some stack overflow code for nwss

covidcast-indicators/nwss_wastewater/delphi_nwss/pull.py

Lines 18 to 30 in a99327b

	def sig_digit_round(value, n_digits):
	"""Round the number of significant digits (x.xxe5 would be 3) to `N`."""
	in_value = value
	value = np.asarray(value).copy()
	zero_mask = (value == 0) | np.isinf(value) | np.isnan(value)
	value[zero_mask] = 1.0
	sign_mask = value < 0
	value[sign_mask] *= -1
	exponent = np.ceil(np.log10(value))
	result = 10**exponent * np.round(value * 10 ** (-exponent), n_digits)
	result[sign_mask] *= -1
	result[zero_mask] = in_value[zero_mask]
	return result

but it would be good to understand if there are good reasons this was written the way it is before simply replacing it.

[melange396]

Good call.

This rounding behavior was asked for in #769 and implemented in #779 (nearly exactly 3 years ago!), but neither the issue nor the PR give hints as to the reason or rationale for the decision. I was around at the time but not involved in this effort, and i cant remember why this might've happened... I have to imagine that it was done for a specific purpose instead of on a whim, but it could have been a premature optimization. Who knows?! Some possibilities for "why" could include:

• to try to save some disk space by shrinking the CSV files a bit?
• a part of the system somewhere else was using a weird representation for these values, like a fixed-point type (in python and/or MySQL), or storing them in strings of a bounded length?
• similar to the above point: potential differences in the default float format or encoding between machine architectures or languages?
• concerns about loss of precision from numerical instability?

[melange396]

A nice intermediate step for this could be to move the sig_digit_round() method into a new delphi_utils module (in https://github.com/cmu-delphi/covidcast-indicators/tree/main/_delphi_utils_python/delphi_utils ).