Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ETA is grossly over-estimated. #77

Open
JenkinsTR opened this issue Jul 25, 2023 · 2 comments
Open

ETA is grossly over-estimated. #77

JenkinsTR opened this issue Jul 25, 2023 · 2 comments

Comments

@JenkinsTR
Copy link

JenkinsTR commented Jul 25, 2023

I think your ETA calculation is off by a factor of 100...

Through both my own manual calculations and visually inspecting the ETA while training, it seems that the decimal separator is placed in the wrong spot.

Example:
23-07-26 02:16:57.959 - INFO: <epoch: 33, iter: 167,900, lr:6.250e-06, t:31.6040s, td:0.0001s, eta:281.8024h> pix-l1: 4.2061e-04 fea-vgg19-l1: 7.0643e-01 l_g_gan: 1.4697e-02 l_d_real: 5.6621e-02 l_d_fake: 5.6456e-02 D_real: 4.0469e+01 D_fake: 3.7594e+01

Here we see that the ETA is 281.8024h(rs), when in fact it is much closer to 2.81hrs. This is consistent with every training run I've done so far, no matter how big or small the ETA actually is.

Even though I've gotten used to it now, I thought I might raise an issue here and let you know 👍

@victorca25
Copy link
Owner

Thanks for letting me know! I'm not currently updating the code, but I'll keep it in mind if I come back to it, I probably missed a factor in the calculation and it displayed ok on my tests, but is incorrect on a different configuration.

Cheers!

@awused
Copy link

awused commented Oct 4, 2023

#60 has been open for two years

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants