New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

多机训练失败后，非master node的进程没有完全kill掉 #416

Open

frankxyy opened this issue Oct 31, 2022 · 3 comments

frankxyy commented Oct 31, 2022

如题，多机训练失败后，非master node还是存活着一个libai进程，导致会持续向控制台打印日志。类似这样的日志：

Collaborator

strint commented Oct 31, 2022

收到，我们尝试复现一下问题。

Flowingsun007 commented Oct 31, 2022

您好，请问【多机训练失败】是手动CTRL + C结束程序，还是代码异常报错失败呢？

我这里基于：https://libai.readthedocs.io/en/latest/tutorials/get_started/quick_run.html 的bert demo跑了一下2机的，CTRL + C以后，master(node0)结束后，node1的程序是可以正常终止的。

Author

frankxyy commented Oct 31, 2022

代码异常报错失败哈

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment