NPC

[kellieotto]

These are some notes for how to simultaneously compute the permutation distribution for each video AND the permutation distribution for the NPC statistic. NB I haven't decided what the NPC test statistic should be - the simplest would be the n^{-1/2} weighted sum of the p-values.

For each video j, j=1,...,Ns, store the permutation test statistic rho's in a vector. We get these from simulate_ts_dist, setting keep_dist = True and iters = B. Put these in a B x Ns matrix.

Make a copy of the rho matrix and sort each column.
For the ith row in the unsorted matrix:
    For the jth column in the unsorted matrix:
        Use numpy.searchsorted OR (rho <= rho[i]).sum to find the percentile of rho_{i,j} --> this is the p-value p-hat_{i,j}
    Compute the NPC for the p-values p-hat_{i,1},...,p-hat_{i,Ns}

From this, we get B NPC test statistics. These are an approximate distribution for the NPC test stat and now we can compare the observed NPC test stat to this distribution to get a p-value.

[kellieotto]

Compare speed/efficiency of numpy.searchsorted versus (rho <= rho[i]).sum()

[kellieotto]

The combining function we're using is invalid. A test combination function must have the following properties. This is taken from the Pesarin & Salmaso book (p 182):

(P.1) A combining function must be non-increasing in each argument. Also, it is generally desirable that is symmetric i.e. invariant with respect to rearrangements of the entry arguments.
(P.2) Every combining function must attain its supremum value, possibly not finite, even when only one argument attains zero.
(P.3) For all significance levels alpha > 0, the critical value of the combined p-values is assumed to be finite and strictly smaller than the supremum of the combining function.

There are several problems with what we're doing: the combining function is a decreasing function of the p-values, not symmetric (since it takes into account sample sizes too), and attains its supremum when all p-values are 1. This might be why we are having trouble interpreting the NSGK results.

It'll be a big task, but I think we can take what we've already done for the non-parametric combination of IRR tests, implement correct combining functions (Fisher's, Liptak's, Tippett's), and convert the npc function to a more general function that can be applied to any tests, not just specific to IRR. It's up and running in permuter, so we can use that as a guide.

[pbstark]

What does the "validity" property give us?

I think what we did gives the right overall significance level--but maybe I'm confused. Do you disagree?

Since the strata are not exchangeable, I don't think symmetry is necessarily desirable; i.e., (P1) seems sideways.

[kellieotto]

I think you're correct that we're getting the right overall significance level. Since the permutation p-values for each test are equivalent to the corresponding test statistics, the combined test stat we use seems like a reasonable thing to do.

However I think it doesn't fit into the "NPC" framework, or if it does I need to think more about why. The authors argue that combining functions that are linear combinations of p-values give rise to inconsistent tests. I also feel a little bit iffy on having both the weights (1/sqrt(sample size)) and the p-values depend on the size of the strata -- seems like double-dipping. If the alternative is true, then the p-value will approach 0 as the sample size increases, so we don't need to weight it by the inverse sqrt of sample size.