Data parallelism technique for Training simplet5 model – CUDA out of memory proplem

[NashaatRJ]

I have an AWS SageMaker instance consisting of 8 GPUs, each with 32 gigabytes of memory. However, when I attempted to train a SimpleT5 model for a text summarization task using high parameter settings, I encountered a CUDA out of memory error as a single GPU with 32GB of memory was insufficient for the task. Could you please help me resolve this issue through converting the training model via data parallelism or any other suitable methods?

Could you please help me resolve this issue through converting the training model via data parallelism or any other suitable methods?