-
Notifications
You must be signed in to change notification settings - Fork 87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Couldn't get the same accuracy with eight commonsense reasoning datasets. #38
Comments
Hi, The set is a little bit different. I listed the commands below. For Series Adapter: For Parallel Adapter: |
ok!I will try these later, thanks a lot |
@ello0211 Hi, did you manage to get the same result as the table reported? Thx! |
Sorry, I didn't conduct the experiment exactly according to the parameters you provided. However, I used LoRa with q and v, taking r=4, and obtained slightly inferior results. By the way, it seems that configuring LoRa as you suggested would result in a large number of parameters, right? |
Hi, with r=32, the number of LoRA parameters should be 8 times of r=4. |
@HZQ950419 I finetuned commonsense_170k.json based on LORA refer to your script, only change eval_step and save_step:
And evaluated by this script:
But still couldn't reproduce the same accuracy as the table. For boolq, only 0.6715 accuracy, and 0.3884 for piqa. Can you help me check the problem. |
Hi, the command is the same as the one we use. Are you using multi-gpu for fine-tuning? Maybe you can try to use single GPU for fine-tuning, as there are some other researchers can't reproduce the results with multi-gpu training. |
Hi,thanks for your great work!
When I try to reproduce the results with commonssense reasoning datasets, it turns out to be not good as the table. The set I use is the same as the math resoning tasks showen in the readme.could you tell me if I use the right set or could you show me the right way to reproduce the same accuracy as the table.
Thank you so much!
The text was updated successfully, but these errors were encountered: