Model saving failed after the first epoch

[wolfshow]

I trained a seq2seq model with 45 million pairs one 4 GPUs. The model was successfully trained for one epoch but crashed during model saving. I would like to know why.

[guillaumekln]

Hello,

Can you share the training logs or at least the error message?

[jsenellart]

hello @wolfshow - did this happen again? can you share the error message? Were you training async or sync on the 4 GPUs?

[jsenellart]

in general, you should considered sampling when dealing with such large input data: one "epoch" will be a subset of your complete dataset (http://opennmt.net/OpenNMT/training/sampling/) - so it will have smaller memory footprint, and you won't risk to lose days of computing.

(but of course it should never crash, but we do need more input here to help)

[wolfshow]

Thanks @jsenellart ! Which sampling method do you recommend to use?

[jsenellart]

Use file sampling, -gsample N - and don't even bother to put all your files together. In a second step, check -sample_dist option to give sampling rules on your collection of training files