[Bug] Fantasy Model Speeds up after 2048 datapoints

[naefjo]

🐛 Bug

Hey all!
We have an online learning setting where we add datapoints to the GP using the get_fantasy_model method. I noticed that there is a significant performance increase (in terms of computation time) when the dataset crosses the 2048 datapoint threshold which I guess can be traced back to #1224 (specifically these lines in linear_operator). Is this still up to date?

I tried recreating the computation time graph in #1224 but I am getting slightly different results where the crossover point is already at around 100 datapoints as compared to 2000. The test was performed in a google colab with a Tesla T4 runtime and locally on a Surface Book 2 with similar results (haven't gotten around to testing this on an RTX 4090 but I assume it will look similar).

I suppose it makes sense to update this value or to create a setting where this parameter can be changed as it is hardware dependent right? I am happy propose a PR for this if you want:)

To reproduce

Code snippet used to generate the plot

import torch
from time import perf_counter
import matplotlib.pyplot as plt

cpu_times = []
gpu_times = []
tensor_sizes = [10,100,200,400,600,800,1000,1500,1750,2000,2500]
for size in tensor_sizes:
  A = torch.rand((size,size))
  time_now = perf_counter()
  torch.qr(A.cpu())
  cpu_times.append(perf_counter()-time_now)
  time_now = perf_counter()
  torch.qr(A.cuda())
  gpu_times.append(perf_counter()-time_now)

plt.semilogy(tensor_sizes,cpu_times, 'x-', label="cpu")
plt.semilogy(tensor_sizes,gpu_times, 'x-', label="gpu")
plt.legend()
plt.show()

Expected Behavior

• get_fantasy_model should not have a sudden decrease in computation time at 2048 datapoints
• i.e. qr decomposition logic should prefer faster option

System information

gpytorch version 1.11
torch version 2.1.0
ubuntu 22.04