Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nan values in training #10

Open
cxym-yyn opened this issue Jul 4, 2022 · 0 comments
Open

nan values in training #10

cxym-yyn opened this issue Jul 4, 2022 · 0 comments

Comments

@cxym-yyn
Copy link

cxym-yyn commented Jul 4, 2022

Hello, thodan.
First of all, thanks for your sharing.
I'm trying to training model with 'python train.py --model='ycbv_custom'', but i got nan values in losses. Like:
step: 0 total_loss: 9.91135 obj_cls: 3.09097505 frag_cls: 4.15891361 frag_loc: 2.40971112
step: 100 total_loss: nan obj_cls: nan frag_cls: nan frag_loc: nan
INFO:tensorflow:global_step/sec: 1.57567
step: 200 total_loss: nan obj_cls: nan frag_cls: nan frag_loc: nan
INFO:tensorflow:global_step/sec: 1.66842

And i check the input data with visualization, it works. I trying to fix for days, But I don't where the error is...
Can you give me some advice for that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant