Potential issue of “errno: 98- Address already in use” error in DDP (with torchrun)

[wakaka9526]

🐛 Describe the bug

During the use of torchrun (with ddp), sometimes there may be random occurrences of ‘errno: 98- Address already in use’, for example:

[W socket.cpp:436] [c10d] The server socket has failed to bind to [::]:29400 (errno: 98 - Address already in use).
[W socket.cpp:436] [c10d] The server socket has failed to bind to 0.0.0.0:29400 (errno: 98 - Address already in use).
[E socket.cpp:472] [c10d] The server socket has failed to listen on any local network address.

and details can see from:
https://discuss.pytorch.org/t/potential-issue-of-errno-98-address-already-in-use-error-in-ddp-with-torchrun/202922

Versions

from torch1.10 ~ torch2.1.0 has been tested, all has the problem.
(the newest torch2.3 has not been tested, but I read the source code, which has no changes. I think the problem still exists in torch2.3)

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang @d4l3k