Regarding Hugginface bitsandbytes issue in inferencing #123866
-
Hello All, I have done a simple project which is Text Summarization and I used T5 Model and I successfully fine tuned my model and also inference it on notebook.i have installed bitsandbytes, accelerate,trl and all. But when push it on hugginface and inference it on server less API. I get error "No package metadata was found for bitsandbytes". Please suggest me how I can resolve this issue. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 6 replies
-
👇👇 The error "No package metadata found for bitsandbytes" means the serverless API can't find the "bitsandbytes" package you used during fine-tuning.
|
Beta Was this translation helpful? Give feedback.
QLoRA for Summarization can benefit from quantization libraries like bitsandbytes. Also u r ryt ,while typical fine-tuning with quantization isn't recommended. But I can provide you with sources to begin or understand :
Also you can explore libraries like post-training quantization . After training, use bitsandbytes to quantize the weights of the trained model for deployment. This reduces model size and imp…