Use scikit-optimize for tuning search space for model selection

[mnarayan]

Create a drop-in replacement for GridSearchCV using scikit-optimize.

The main benefit here is that scikit-optimize can ensure that not all points in the search space or grid are evaluated as is done by GridSearchCV. Currently QuicGraphLassoCV is modeled after GridSearchCV while taking advantage of path mode of the solver. We would instead need to model this estimator class after BayesSearchCV

Similarly, other model selection criteria that involve searching along a regularization path (as opposed pure model ensembling) could also have analogues such as BayesSearchEBIC and so forth.

[mnarayan]

This is how one would invoke BayesSearchCV using QUIC's default mode.

    model = BayesSearchCV(QuicGraphLasso(init_method='corrcoef', score_metric=metric),
                         cv=num_folds,
                         n_iter=16,
                         refit=True
                         ) 
    model.add_spaces('space_1', {
                        'lam': Real(1e-02,1e+1, prior='log-uniform')
                        }
                    )
    model.fit(X)

Here is an example of using BayesSearchCV

Results from running run skggm_optimal_selection_example.py output

QuicGraphLasso + GridSearchCV with:
   metric: kl
   len(cv_lams): 100
   cv-lam: 0.148496826225
   lam_scale_: 1.0
   lam_: 0.148496826225
QuicGraphLasso + skopt.BayesSearchCV with:
   metric: kl
   Final score: -0.224238139648
   cv-lam: 0.139651618375
   lam_scale_: 1.0
   lam_: 0.139651618375

[jasonlaska]

Based on your example here, this feels more like something that should go in /examples and not something that should be part of the main package. My reasoning is:

• we should shy away from providing light wrappers over other packages (or just combinations of packages) since then we have increased maintenance when they change their interface, introduce incompatibilities, etc. It's fine to provide an extensive set of examples using these, but otherwise, where would we draw the line in terms of combinations of helper interfaces.

• if we needed to specially modify the QuicGraphLasso to take advantage of this package (such as the more optimized QuicGraphLassoCV), that would be different, but based on your example and the way BayesSearchCV works, it doesn't seem that we need to.

• I do think that a rich set of examples on how to use the tools and best practices makes sense, but it need not be on the mainline package.

[mnarayan]

@jasonlaska The example I made was just a brute force example to show usage of BayesSearchCV which is a wrapper around skopt's Optimizer class.
The real changes I think need to happen as you pointed out here

• if we needed to specially modify the QuicGraphLasso to take advantage of this package (such as the more optimized QuicGraphLassoCV), that would be different, but based on your example and the way BayesSearchCV works, it doesn't seem that we need to.

skopt's Optimizer lets you go about adaptively picking a grid of points to evaluate. For example, this would not be possible to achieve to accomplish for EBIC by invoking BayesSearchCV. We would need to model our own QuicGraphLassoCV and QuicGraphLassoEBIC similar to BayesSearchCV. Presumably, we could analogously take advantage of path mode as well, just like you enabled with QuicGraphLassoCV.

The moment we have more than one path to optimize over multiple penalties or multiple parameters, this starts to become a lot more handy as well.