[Feature Proposal] Distributed Training for FL

[slyviacassell]

As the title described, does standalone mode support multiple GPUs to speed up training?

[dunzeng]

We didn't provide multiple GPUs in the standalone module.
However, you can use the DP module of PyTorch in train function in SGDSerialClientTrainer.

[slyviacassell]

We define the following variables to further illustrate the idea:

• K: the number of clients who participated in training each round
• N: the number of available GPUs

When K == N, each selected client is allocated to a GPU to train.

When K > N, multiple clients are allocated to a GPU, then they execute training sequentially in the GPU.

When K < N, you can adjust to use fewer GPUs in training.

We need to set the number of GPUs in gpu and specific distributed settings in the distributed configs.

The implementation is under working. Anybody would like to help?