NaN or Inf found in input tensor

[xiaozhouguo94]

I trained model on Aishell corpus, and I found a error in train log: NaN or Inf found in input tensor.

So, how dose it come about? How does it affect the outcome. Thank you.

By the way, your work is excellent !

[hirofumi0810]

@xiaozhouguo94 What kind of model did you use? Can you show me the error log?

[xiaozhouguo94]

@xiaozhouguo94 What kind of model did you use? Can you show me the error log?
I use examples/aishell/s5/conf/asr/blstm_las.yaml,
and error log is:
2020-12-15 05:28:59,007 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:28:59,007 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:28:59,008 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:28:59,222 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:28:59,519 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:28:59,520 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:28:59,520 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:28:59,744 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:28:59,925 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:28:59,926 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:28:59,927 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:29:00,143 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:29:00,332 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:29:00,337 root line:14 WARNING: NaN or Inf found in input tensor.
2020-12-15 05:29:00,338 root line:14 WARNING: NaN or Inf found in input tensor.

I just paste a little bit, and there are lots of error in train log. Even these error is reported, the train process will still continue, and final cer is so terrible, just like 5544.05%. I found this error start to appear in epoch 5, and Neural SP would also finally save the model.epoch-5, rather than the result after the last epoch(model.epoch-25). So I think Neural SP found this error and didn't trust the training result after epoch 5. I'm confused that how to solve it?
Thanks you

[hirofumi0810]

@xiaozhouguo94 Let me check it.