Optimization that might unlock real-time responsiveness - possible?

[wolfgangmeyers]

So I have a hypothesis - I think that the maximum speed of Big Sleep is determined by how fast data can travel back and forth between CPU and GPU memory. I captured what my GPU usage looks like on an RTX 3090:

It consumes a fair bit of video memory, but on average around 4% of the processing power. When I tried running multiple processes, the iterations / second dropped for each process so that the same amount of work was being done in total per second.

Does anyone else have thoughts on this? If I'm right about this, is it possible to keep the data in the GPU? I think it could massively speed up the processing speed, and using it could be more of a real-time experience.