Value error

[ImmortalSdm]

I'm trying to train ComodGAN on FFHQ. However, i'm facing with the ValueError problem below.
Traceback (most recent call last): File "run_training.py", line 134, in <module> main() File "run_training.py", line 129, in main run(**vars(args)) File "run_training.py", line 72, in run dnnlib.submit_run(**kwargs) File "/home/dmsheng/demo/image_inpainting/co-mod-gan-default/dnnlib/submission/submit.py", line 343, in submit_run return farm.submit(submit_config, host_run_dir) File "/home/dmsheng/demo/image_inpainting/co-mod-gan-default/dnnlib/submission/internal/local.py", line 22, in submit return run_wrapper(submit_config) File "/home/dmsheng/demo/image_inpainting/co-mod-gan-default/dnnlib/submission/submit.py", line 280, in run_wrapper run_func_obj(**submit_config.run_func_kwargs) File "/home/dmsheng/demo/image_inpainting/co-mod-gan-default/training/training_loop.py", line 358, in training_loop metrics.run(pkl, run_dir=dnnlib.make_run_dir_path(), data_dir=dnnlib.convert_path(data_dir), num_gpus=num_gpus, tf_config=tf_config) File "/home/dmsheng/demo/image_inpainting/co-mod-gan-default/metrics/metric_base.py", line 188, in run metric.run(*args, **kwargs) File "/home/dmsheng/demo/image_inpainting/co-mod-gan-default/metrics/metric_base.py", line 82, in run self._evaluate(Gs, Gs_kwargs=Gs_kwargs, num_gpus=num_gpus) File "/home/dmsheng/demo/image_inpainting/co-mod-gan-default/metrics/inception_discriminative_score.py", line 69, in _evaluate s, _ = scipy.linalg.sqrtm(np.dot(sigma_fake, sigma_real), disp=False) File "/home/dmsheng/anaconda3/envs/tf1.14/lib/python3.6/site-packages/scipy/linalg/_matfuncs_sqrtm.py", line 161, in sqrtm A = _asarray_validated(A, check_finite=True, as_inexact=True) File "/home/dmsheng/anaconda3/envs/tf1.14/lib/python3.6/site-packages/scipy/_lib/_util.py", line 263, in _asarray_validated a = toarray(a) File "/home/dmsheng/anaconda3/envs/tf1.14/lib/python3.6/site-packages/numpy/lib/function_base.py", line 486, in asarray_chkfinite "array must not contain infs or NaNs") ValueError: array must not contain infs or NaNs
I've tried rebuilding the tfrecord and several hyper-parameters, no work. Any ideas?

[zsyzzsoft]

What is your tf version?

[ImmortalSdm]

What is your tf version?

1.14.0