[RFC, WIP]: Remove evaluate kernel

[gpleiss]

With KernelLinearOperator defined in the linear operator package, LazyEvaluatedKernelTensor becomes kind of redundant.

This PR attempts to improve the lazy evaluation of kernels in GPyTorch, by doing the following:

• Remove LazyEvaluatedKernelTensor, instead using KernelLinearOperator (which is also used by KeOps kernels)
• Removing evaluate_kernel form GPyTorch (which will be redundant with KernelLinearOperator) and also removing it from the LinearOperator package.

cc/ @Balandat

[jacobrgardner]

Aren't we going to take at least some amount of performance hit here on kernels you don't lazily evaluate when making predictions, because we don't get to do indexing before calling the kernel?

[Balandat]

So being able to use self.batch_shape here relies on the fact that all parameters have a fully explicit shape (rather than broadcasting over them)?

[gpleiss]

Yes. I think this is generally true for all kernels in GPyTorch. We can add something to the documentation to be more explicit about this.

[Balandat]

We're pretty deep in the indentations here, might make sense to pull out some of the above into helper functions to make it clear what's going on here / easier to read the code

[Balandat]

Changing the parameter shapes for widely used kernels like this will likely result in a bunch of downstream backward compatibility issues. I'm not necessarily suggestion not to do this, but those changes will need to be properly documented / announced.

[Balandat]

btw, if we're doing this, we should probably combine this with updating the shapes of the priors (c.f. #1317) to make all of that consistent

[gpleiss]

I'm going back and forward about changing parameter shapes. On one hand, it could be quite the breaking change. On the other hand, it would create more consistency, and we should also be able to dramatically simplify many kernels as well.

@Balandat thoughts? I imagine this would have the biggest impact on BoTorch.

[gpleiss]

Actually, what probably makes the most sense is leaving kernel parameter shapes the way they are currently, and potentially creating consistent parameter shapes and addressing #1317 in a separate PR.

[Balandat]

Makes sense. I think fixing the parameter shape inconsistencies and making things consistent across the board would be good, and hopefully this would also get rid of some of the long standing bugs/issues. This would impact botorch, but if we coordinate on the releases we can make sure that the effect of this is minimized. It would likely generate some short term pain for some power users (of both gpytorch and botorch) with custom setups, but I think that short term pain is probably the long-term gains from consistency.,

[gpleiss]

@Balandat That sounds like a good plan. Here's what I'm thinking as a course of action:

• Make a PR that makes kernel parameters consistent sizes (and adds appropriate warnings/deprecations/etc)
• Address [Bug] Sampling from priors doesn't match shape of hyperparameters #1317
• Merge this PR
• (+ other breaking changes that we've wanted to address for a while)

After all of that, we make a major release, timed with a BoTorch release.

[gpleiss]

Aren't we going to take at least some amount of performance hit here on kernels you don't lazily evaluate when making predictions, because we don't get to do indexing before calling the kernel?

@jacobrgardner I don't think it'll lead to a loss in performance. This PR mostly merges LazyEvaluatedKernelTensor and KeOpsLinearOperator, since they share most of the same functionality.