Fixing some error metrics in profiling and network simulations

[mnarayan]

• Frobenious norm sometimes counts diagonal info (like in Average Power). Make sure this is changed across the package.

error_fro = np.linalg.norm(np.triu(adj - new_estimator.precision_,1), ord='fro')

• Numerical issues in model selection error with banded networks.
  Why do these give inconsistent numbers, particularly exact_support versus count_support_diff

print 'Exact Recovery (T/F): {}, (TPR,FPR): {}, Count Diff: {}'.format(
    exact_support(adjacency,prec_hat),
    approx_support(adjacency,prec_hat,prob=0.1)[1:2],
    count_support_diff(adjacency,prec_hat),
)

• Issues in [approx_support().] To be fixed by re-using other functions from AveragePower

[jasonlaska]

Numerical issues with exact_support and approx_support persist in the .profiling submodule. It's real weird because I have tests on these metrics inverse_covariance/profiling/metrics and they pass.

https://github.com/skggm/skggm/blob/develop/inverse_covariance/profiling/tests/metrics_test.py#L187-L227

[mnarayan]

@jasonlaska Does this mean we still have to do some artificial thresholding? I'll take a look at this again too.

[jasonlaska]

As you play around with the tools, please advise and I'll integrate fixes in.

[mnarayan]

So I have at least one potential clue regarding numerical issues. They might be related to using 3-fold CV.

Using cv=(3,10) gives the following support

array([  1.04618665e+00,   0.00000000e+00,   0.00000000e+00,
         0.00000000e+00,   0.00000000e+00,   8.41176079e-04,
         0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
        -2.19816011e-01])

Whereas using cv=(2,10) gives the following

array([ 1.04241831,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        , -0.21027987])

This makes sense as most regimes for covariance estimation, 3-fold results in too little data for model re-fitting in hold out and creates more false positive selections. The values are tiny though, however. We could consider changing the default fold number to 2 or adding in an example to show that unlike prediction problems, we might be more sensitive to choice of fold number.