The performance gap between using vanilla 3dVAE and Casual3dCNN

[Sutongtong233]

Hi, I am interested in the performance of using 3d CNN without casual for VAE training, and corresponding results of generation. As I understand, the propose of Casual3dCNN is for autoregressive model of stage two in MagViT. Since our stage two is still diffusion, is there any consideration of using Casual3dCNN?

[LinB203]

We are currently just inflating 2DVAE to 3DVAE and replacing conv3d with causalconv3d. this already supports simultaneous training of images and videos.