New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

在训练的时候出现了这个错误，Found Inf or NaN global norm，哪个大牛帮忙看一下怎么破？ #25

Open

guang1234 opened this issue Jan 24, 2019 · 1 comment

Labels

help wanted question

guang1234 commented Jan 24, 2019

InvalidArgumentError (see above for traceback): Found Inf or NaN global norm. : Tensor had Inf values
[[node VerifyFinite/CheckNumerics (defined at C:\Users\xxx\seq2seq-couplet-master1\model.py:79) = CheckNumericsT=DT_FLOAT, message="Found Inf or NaN global norm.", _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

guang1234 changed the title ~~在训练的时候出现了这个错误，哪个大牛帮忙看一下怎么破？~~ 在训练的时候出现了这个错误，Found Inf or NaN global norm，哪个大牛帮忙看一下怎么破？

wb14123 added help wanted question labels

Owner

wb14123 commented May 5, 2022

Try to restart the training from checkpoint

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment