GSoC_2016_project_large_gps

Large-Scale Gaussian Processes

Polish, update, and extend Shogun's framework for Gaussian Processes, with a focus on large-scale problems and sparse variational inference.

Mentors

• Wu Lin (github: yorkerlin, IRC: yorkerlin)
• Emtiyaz Khan
• James Hensman

Difficulty & Requirements

Medium

You need know:

• Gaussian Process basics (You should understand the GP Notebook)
• Variational approximation basics (You should understand at least the full GP part of the Notebook for Variational Inference)
• Linear Algebra (math & in C++)

Description

Following our previous successful project on variational learning for Big Data, we attempt to bring Shogun's Gaussian Processes (GP) to Big Data land. From a high level perspective, this means that the goal is to implement established methodology on how to scale up GPs to be able to process hundreds of thousands of points.

Details

We want to focus on the following sub-tasks:

• Variational inference for (full) GP
• Variational inference for sparse GP
• Stochastic variational inference for sparse GP
• Applications

The project will start from the existing code base, which already contains a huge amount of work. We aim to fill the gaps in the above methods:

• Implement all standard methods that are not yet in the framework
• Benchmark existing methods against competing implementations (such as GPStuff, GPflow, GPML, etc)
• Improve efficiency if necessary. Make explicit use of multi-core computation.
• Unify/Refactor gradient computations within Shogun (Note that the new framework should be easy to be extended to using AutoDiff)

Waypoints and initial work

• do benchmarks about all existing inference methods for (full) GP and sparse GP under the existing framework (entrance task) (KLCovarianceInferenceMethod, KLCholeskyInferenceMethod, KLApproxDiagonalInferenceMethod, FITCInferenceMethod, and SparseVGInferenceMethod) See #3054

• Understanding the inference procedure and refactoring existing framework with the help from mentors

• Make beautiful demos and benchmarks under the new framework

• Stochastic variational inference for GPs (optional)

Refactoring existing framework

Variational Gaussian inference (Suggested Roadmap)

• base class for computing gradient of Evidence Lower BOund (ELBO) wrt variaitonal variables
• base class for computing gradient of ELBO wrt hyper-parameters in likelihoods, mean functions, and co-variance/kernel functions
• (for full GP) classes for computing gradient wrt variaitonal variables using existing hand-implemented codes
• (for full GP) classes for computing gradient wrt hyper-parameters using existing hand-implemented codes
• Benchmarks and notebooks for demos
• base class for MC samplers
• classes for using existing MC samplers
• (for sparse GP) classes for computing gradient wrt variaitonal variables using existing hand-implemented codes
• (for sparse GP) classes for computing gradient wrt hyper-parameters using existing hand-implemented codes
• classes for HMC samplers from Stan (optional)
• base class for model selection (eg, Bayesian OPT) (optional)

Optional

• MCMC inference
• Deep GP

Why this is cool

Our primary goal is to scale up GPs to use them in "big data" applications. GPs are becoming more and more popular for big data because they provide both accurate predictions, tell us how confident we should be about our prediction (aka uncertainty quantification) and also whether we have selected the right model (aka model selection). With big data comes big noise, so these issues are becoming ever more relevant. Recently GPs are a popular choice beyond regression and classification for a wide range of appliations from numerical optimization to recommendation system and even to deep networks. The main bottleneck in these applications is scalability. Therefore, we want you to write easy-to-use, scalable code which will help the machine learning community to use GPs.

Useful ressources

• GPflow using Tensorflow
• Shogun's variational Gaussian inference for full GP
• Shogun's variational Gaussian inference for sparse GP
• Notebook about Variational inference for GP
• Approximations for Binary Gaussian Process Classification
• Concave Gaussian Variational Approximations for Inference in Large-Scale Bayesian Linear Models
• Variational Learning of Inducing Variables in Sparse Gaussian Processes
• Gaussian Processes for Big data
• Scalable Variational Gaussian Process Classification
• Distributed Gaussian Processes
• Deep Gaussian Processes

Get back to the main projects page.