Need help understanding the implementation of training with LoRa

[dangnguyenngochai]

I have the following two questions and would very appreciate if anyone can help:

1. In the paper, you stated that you fine-tuned Llama 2 with LoRa. From what I understand, the base model and the adapter from Lora need to be loaded separately and combined using merge_and_unload from this link. However, I can not seem to see the part of the code in your codebase ?

2. Along with that, I am having a problem running the demo with the 4-bit quantization by change the load_in_4bit to load_in_8bit,. Here is the error:

Traceback (most recent call last):
File "/content/private_scripts/MiniGPT4-video/minigpt4_video_inference.py", line 179, in
pred=run(video_path,instruction,model,vis_processor,gen_subtitles=add_subtitles)
File "/content/private_scripts/MiniGPT4-video/minigpt4_video_inference.py", line 128, in run
answers = model.generate(prepared_images, prompt, max_new_tokens=args.max_new_tokens, do_sample=True, lengths=[length],num_beams=1)
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/content/private_scripts/MiniGPT4-video/minigpt4/models/mini_gpt4_llama_v2.py", line 553, in generate
outputs = self.llama_model.generate(
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/peft/peft_model.py", line 580, in generate
return self.base_model.generate(**kwargs)
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/transformers/generation/utils.py", line 1525, in generate
return self.sample(
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/transformers/generation/utils.py", line 2622, in sample
outputs = self(
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
File "/content/private_scripts/MiniGPT4-video/minigpt4/models/modeling_llama_v2.py", line 66, in forward
outputs = self.model(
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 1070, in forward
layer_outputs = decoder_layer(
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 798, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 386, in forward
query_states = self.q_proj(hidden_states)
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/envs/minigpt4_video/lib/python3.9/site-packages/peft/tuners/lora.py", line 348, in forward
result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (3030x4096 and 1x8388608)