1D Statistics

[kanitw]

Goal

• Refactor API such that we can do calculate separately and pass these as input to our API
  • If the stats are not passed as input, then we recalculate internally.
  • If they are already input, then they are good!

Basic Implementation

• extend DlFieldProfile so that we can store more statistic like ones below

Stats

• Number of potential outliers

  • Easy -- just run a filter with quartiles numbers
    • Tukey's method -- https://en.wikipedia.org/wiki/Outlier#Tukey.27s_test
  • maybe store index

• Entropy (for measuring uniformity)

  • Easy -- just call dl.entropy

• Number of Unique Values (We already got this from dl.profile)

Rank by Feature http://ieeexplore.ieee.org/document/1382892/

(RBF)

• Normality
• Size of the biggest gap