New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow NaN or None values to be passed in, and silently ignored #90
Comments
You can solve this by calling Afterwards, you can just pass df to tsfresh |
This was a design decision. tsfresh will not tinker with the input time series data by for example imputing values or dropping NAs. Reason behind this: In data science projects, NAs should be handled with special care, often they contain a lot of information. We don't want our packages to silently remove those informations by dropping it. This is way we are not imputing the input data. |
In many cases doesn't it make sense to treat nans as nans, rather than requiring imputation? For example, features like mean, max and min are more informative if the nans are simply ignored when computing the value. |
Actually, there is a workaround by customizing features. There is, however, a function I discovered that is not consistent with the design principle introduced in this post when doing so. This is confirmed with dummy data:
A fix would be simply set the param to
|
In my DataFrames, I oftentimes find it very reasonable to have quite a few NaN or None values.
In order to run tsfresh, I just set all those values to 0, which is... not ideal. Even imputing missing values would not work particularly well for several of my use cases.
Yet, it seems (from a super naive outsider's perspective), like this is filtering that tsfresh could do relatively easily itself.
When it grabs each time_series, it can simply remove or categorically ignore NaN/None, and compute features on the values that do exist. This makes my life easier when, say, one customer signs up a month after another customer, and thus has missing values for that month.
Again, super naive outsider perspective here, I know this might be impossible. But if it is possible, I'd love to add in that bit of filtering!
The text was updated successfully, but these errors were encountered: