Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fine-tune out of memory #11

Open
jiangix-paper opened this issue Feb 13, 2023 · 3 comments
Open

fine-tune out of memory #11

jiangix-paper opened this issue Feb 13, 2023 · 3 comments

Comments

@jiangix-paper
Copy link

jiangix-paper commented Feb 13, 2023 •
edited

请问使用这个参数fine-tune，显存占用大概多少？我这边16G的v100显示out of memory

image

请问如果单卡fine-tune显存不够用，有多卡的fine-tune代码吗
@joytianya
Copy link
Collaborator

可以调整batch size和max len等参数使用

@yzhzx
Copy link

yzhzx commented Jun 6, 2023

batch size 调成4 占用不到11G显存可以完成训练 , 就是训练完成使用没有示例的效果 就不知道怎么调整了

@joytianya
Copy link
Collaborator

可以调小学习率，训练开始，训练一段测试一下，比如训练100步，如果有问题，训练10步看看，看看是不是训练中的问题还是推理的问题

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yzhzx @joytianya @jiangix-paper