Llama 3 8B GPU requirements?

[Leonard907]

Thanks for the integration for Llama 3 models. I'm interested in fine-tuning Llama 3 8B with full 8K context using LoRA. What resource requirements are needed? Can I do it in just 1 A100 80GB GPU?

[ebsmothers]

Hi @Leonard907 thanks for creating the issue! The exact resource requirements depend on batch size and other training configs, but if I run for instance

tune run lora_finetune_single_device --config llama3/8B_lora_single_device \
dataset=torchtune.datasets.slimorca_dataset dataset.max_seq_len=8192

I am able to fit pretty comfortably into a single A100 80GB.

[Leonard907]

Thanks!