Validation and testing too slow when train Transformer

[ymjiang]

My script for training Transformer:
MXNET_GPU_MEM_POOL_TYPE=Round python train_transformer.py --dataset WMT2014BPE --src_lang en --tgt_lang de --batch_size 7000 --optimizer sgd --lr 2.0 --warmup_steps 4000 --save_dir transformer_en_de_u512 --epochs 30 --gpus 0,1,2,3,4,5,6,7 --scaled --average_start 5 --num_buckets 20 --bucket_scheme exp --bleu 13a --log_interval 10

The training goes well, like this: (training one epoch takes about 1.5 hour)
[Epoch 0 Batch 50/2853] loss=10.5437, ppl=37939.2041, throughput=70.37K wps, wc=942.95K

However, when one epoch ends, the validation and testing process are very slow. For example, it takes about 30-40 minutes to finish the validation process (3000 samples for validation in total).

Any body has idea? Perhaps it is due to something missing, but I am not sure.

---

I am using:

• Gluonnlp 0.5.1 (commit f4275c0)
• MXNet 1.3.1
• python 2.7

My platform info:

• 64 CPU Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
• 200 GB memory
• GPU: 32GB V100 (utilization ~50% during training)
• Isolated environment (as Docker container, but I ensure no other process is running on the same machine)
• CUDA 10.0

[szha]

@ymjiang 30-40m for validation definitely sounds abnormal. Are you using gluonnlp 0.5.0 when running the script? Would you mind trying the version on master branch, with mxnet nightly?

Could you share a bit more on the platform on which you run the experiment?

1. GPU, CPU, RAM size
2. Whether it's a shared environment
3. CUDA version

[szha]

cc @eric-haibin-lin @szhengac in case they have other suggestions.

[ymjiang]

@szha Thanks for the quick response. I am using gluonnlp 0.5.1, cloned from master branch (f4275c0). I will try gluonnlp 0.5.0.

I updated my platform info in the description.

[eric-haibin-lin]

Training throughput also seems low. What's your GPU utilization during training?

[ymjiang]

@eric-haibin-lin The utilization is about 50% for each GPU during training.

What should be the ideal throughput? And by the way, I set num_accumulated to 1. Could this be the reason that the throughput is low?

[szhengac]

To achieve good performance on transformer, a large batch is typically needed, so we set num_accumulated to 16 for the pretrained model. Batch size 7000 per GPU may be too large as some buckets may not have enough samples to support this number and therefore cause imbalance. Thus, I would recommend using larger num_accumulated instead of batch size. In the first 1-3 epochs, the testing is slow, as the prediction is not accurate and it would take a lot of steps to get EOS token or stop until it reaches the maximum length. When the training proceeds, you will find that the testing takes less time. The final model typically takes 4-5 minutes to finish the testing.

[szhengac]

On one 8GPU V100 machine, it is supposed to get 160k wps.

[ymjiang]

@szhengac Thank you very much for the detailed explanations. Let me try to set larger num_accumulated and see if it works as you described.

[ymjiang]

Problem solved, I hard-code the optimizer to SGD in train_transformer.py for some reasons. When I use Adam instead, the loss decrease much more quickly. At final step, the validation only takes about 5 minutes. And the training speed reaches 130 kwps. Sorry about my mistake.