[Feature Request]: Nan Values for correlation and cross correlation

[Jeanselme]

Would it be possible to automatically ignore the nan values when computing np.corrcoef or np.correlate ? We could create a function like np.nan_correlate.

In the case of corrcoef it is straight forward and can be solved by ignoring the nan values of both arrays, however in the convolution setting, it might have different lag on the two series which would create unwanted results.

Current Behavior

import numpy as np

a = np.array([0, 1, 2, 3, 4])
b = np.array([0, 1, np.nan, 3, 4])

np.correlate(a, b, 'full')

Returns array([ 0., 4., nan, nan, nan, nan, nan, 4., 0.])

It would be useful in some case to return :
Returns array([ 0., 4., 11., 18., 26., 18., 11., 4., 0.])

This is just ignoring any nan in the summation

[Harry-Kwon]

I'd like to work on adding an optional argument to those functions to ignore NaN values.

[aleksejs-fomins]

First of all, I would appreciate this functionality as a user. Thank you for bringing this up

Secondly, please be careful when implementing this. For many applications, the interesting quantity is not the correlation coefficient itself, but the mean correlation coefficient corr(x, y) / len(x). So far, the users have manually removed nan's before processing, which is hard, but correct. Now, if a user has nan's in the data and they are implicitly dropped by the correlate function, the user might proceed to unsuspectingly divide the correlation by len(x), whereas they should only be dividing by the length of the non-nan part of the sum. So firstly, I suggest that correlate should throw a warning if there are Nan's in the arguments. Secondly, perhaps it makes sense to implement a mean correlation function (e.g. correlate_mean(x,y,...) ), which would divide the overlap by its non-nan length.

[rossbar]

The addition of more nan* functions has been discussed and the current consensus is against doing so, so I will close this for now. If you are interested in pursuing the feature request, consider bringing it up on the mailing list (you can link to the issues for context).