New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

全参数微调需要显存？以及多卡模型分布的问题。 #314

Open

hekaijie123 opened this issue May 15, 2024 · 1 comment

hekaijie123 commented May 15, 2024

你好。我有两张A100 (40G)显存。

但是我运行全参数微调时batchsize=1也会超显存，全参数微调需要单张卡的显存多少？有没有办法将模型分不到两张卡上做训练？
我想做某个领域知识的预训练，全参数微调是否可行，需要多少量级的数据。如果全参数微调不可行，那么是否可以多阶段lora微调？

Author

hekaijie123 commented May 15, 2024

@myownskyW7 @LightDXY @lvhan028

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment