Kronecker-factored matrices

[willtebbutt]

Consider a kernel of the form

(k)(x::Vector, x′::Vector) = prod(map(k.ks, x, x′))

where k.ks is a collection of kernels which can be applied to single-dimensions, and the dimensionality of x and x′, and the length of k.ks is D.
If we wish to evaluate the covariance matrix associated with a matrix of inputs which can be represented as a Cartesian product:

X = IterTools.product(x1, ..., xD)

where each xd is a vector, then cov(k, X) can be represented as the Kronecker product of a set of D matrices, the dth of which is cov(k.ks[d], xd). If this structure is present then the asymptotic complexity of inference is greatly reduced.

• Details of this can be found in [1].
• There was work at NIPS 2017 in the AABI workshop which maybe relates to this work - see [2].
• There is no obvious reason why one shouldn't be able to exploit matrices which admit partial Kronecker factorisations. I'm not sure what / if there is relevant work is here though.
• Perhaps there is something in the tensor decomposition / tensor train field / line-of-work (I'm not even sure what to call this) that can speak to the above. This video looks related: https://www.youtube.com/watch?v=CrSFne4h3Pk

To achieve efficient inference here, it will be necessary to implement a Kronecker-factorised matrix type which has appropriate efficient representations.

[1] - Saatçi, Yunus. Scalable inference for structured Gaussian process models. Diss. University of Cambridge, 2012.
[2] - Evans, Trefor W., and Prasanth B. Nair. "Scalable Gaussian Processes with Grid-Structured Eigenfunctions (GP-GRIEF)."

[willtebbutt]

Kronecker.jl is a thing

[MichielStock]

This is a large body of research using Kronecker factorizations in GP/kernel ridge regression, e.g. GPatt and Kronecker ridge regression. Would love to discuss this or contribute!

[willtebbutt]

Would be great to both discuss and collaborate :) Are you around for the hackathon tomorrow?

[MichielStock]

Yes! I am here the whole Hackathon. Would like to get involved 🙂

…

On Thu, 25 Jul 2019, 16:48 willtebbutt, ***@***.***> wrote: Would be great to both discuss and collaborate :) Are you around for the hackathon tomorrow? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABN5PI5BN66TCIGXVLFMFB3QBIGQXANCNFSM4E5OIFTA> .