[FEATURE] Add DoRA: Weight-Decomposed Low-Rank Adaptation

[pascal-pfeiffer]

🚀 Feature

Add support for Weight-Decomposed Low-Rank Adaptation (DoRA).
"DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. "

Paper: https://arxiv.org/abs/2402.09353
Implementation in Peft: https://github.com/huggingface/peft/releases/tag/v0.9.0

Motivation

Authors claim that DoRA is more efficient w.r.t. param count as LoRA and achieves better results throughout all tests.

[tmostak]

I'd like to bump this... seems DoRA yields accuracy basically on par with full fine tuning.