last.ckpt or best.ckpt, which is better?

[branlsnyder]

Hi All,

When resuming a training, is it recommended to resume from the best.ckpt or the last.ckpt? Occasionally I also see a last-v1.ckpt.

So far, I have always resumed from last.ckpt. But I noticed today that my reconstructions sounded really different depending on whether I used last.ckpt, or best.ckpt.

Brandon

[caillonantoine]

Hi ! Last is the last checkpoint saved, v1 means it is an intra epoch checkpoint, best is the checkpoint with the best validation loss ! Usually I recommend using the last checkpoint as the validation loss only is valid for the first phase :)

[branlsnyder]

great! thanks for the clarification.

[asantos-6]

Hey! What about when I have trained a model for several days, but the "best.ckpt" file predates the "last.ckpt"? In that case, the 'best' is the optimal model and we should export it instead of the 'last' right?