Abut the different of code and paper

[Yi-Qi638]

The paper claims that ODISE freezes the Denoising Unet. However, upon inspecting the code from ODISE's "ldm.py" file, I encountered some aspects that left me uncertain about the actual freezing status of the Unet. This code is from ODISE/odise/modeling/meta_arch/ldm.py 974

[xvjiarui]

Hi @wangyiqi
Since we need to back propagate the gradient through UNet ( we are not changing the UNet feature, we just need the grad to optimizer the MLP of implicit captioner), so we need to set the grad before we forward.