Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JP-3250: Exclude entire var if any NaN present #7711

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

tapastro
Copy link
Contributor

@tapastro tapastro commented Jul 7, 2023

Resolves JP-3250
Closes #7647

This PR addresses an issue found when a variance array contains NaN values - the sum with the other variance arrays leads to NaN values in the error arrays. The current behavior will propagate any NaNs in a variance array to the final error array.

The approach in this PR is to fully exclude a variance array if at least one NaN is present - this is to prevent the error array from having a variable number of contributions. Alternatively, one could let the error array exclude the NaN values from summation for individual variance entries, leading to the error having 0-3 variance contributions depending on the number of NaNs present.

Checklist for maintainers

  • added entry in CHANGES.rst within the relevant release section
  • updated or added relevant tests
  • updated relevant documentation
  • added relevant milestone
  • added relevant label(s)
  • ran regression tests, post a link to the Jenkins job below.
    How to run regression tests on a PR
  • Make sure the JIRA ticket is resolved properly

@tapastro tapastro added this to the Build 10.0 milestone Jul 7, 2023
@tapastro tapastro self-assigned this Jul 7, 2023
@tapastro tapastro requested a review from a team as a code owner July 7, 2023 18:34
@izkgao
Copy link
Contributor

izkgao commented Feb 2, 2024

Would np.nansum be a better option?

import numpy as np
np.nansum([np.nan, 1.0, 2.0])
# 3.0
np.nansum([np.nan, np.nan, np.nan])
# 0.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

NaNs in the flat variance array contaminate the x1d combined error estimate
2 participants