Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

recon images is black #31

Open
syyxsxx opened this issue Jan 16, 2024 · 9 comments
Open

recon images is black #31

syyxsxx opened this issue Jan 16, 2024 · 9 comments

Comments

@syyxsxx
Copy link

syyxsxx commented Jan 16, 2024

i train magvit2 on imagenet
the recon images is black, but the loss looks normal
sampled 18 (1)
image

@nangying1112
Copy link

I met the same problem. Have you solved this?

@dyflional10
Copy link

I met the same problem. Have you solved this?
I also encountered the same problem where the loss remains constant, and I discovered that the gradient is NaN.

@syyxsxx
Copy link
Author

syyxsxx commented Jan 22, 2024

I turned off mixed precision training,solve this problem

@lucidrains
Copy link
Owner

@syyxsxx thank you for finding and sharing the source of the issue. i'll figure out what's the matter with mixed precision..

@dyflional10
Copy link

@lucidrains Thanks for your great work. I turned off mixed precision training, the loss looks normal, but the image reconstruction is abnormal.
sampled 273
截屏2024-01-24 11 00 03

@misery0424
Copy link

misery0424 commented Jan 24, 2024

I turned off mixed precision training,solve this problem

can you share your config and training log, thanks

@VoyageWang
Copy link

I also met the same problem the construction image is not so normal. Can you share the details of solving this problem? How to turn off the mixed precision traning

I turned off mixed precision training,solve this problem

@J-Wu97
Copy link

J-Wu97 commented Feb 26, 2024

我在 imagenet 上训练 magvit2, 侦察图像是黑色的,但损失看起来很正常 采样 18 (1) 采样 18 (1) 图像

hello,Have you solved this problem?

@kxgong
Copy link

kxgong commented May 16, 2024

Hi, I have a question. Why use images to train magvit-2? Should it designed for videos?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants