Hardware requirements for training and development

[pipoupiwam]

Hi everybody!
First of all thank you for your amazing work, I discovered Open Assistant yesterday on youtube, and am very impressed !

I would like to experiment with Open Assistant on my local setup, i browsed the repository and issues for hardware requirements ( VRAM ).

What are the VRAM requirements for running the model ?
Would a 4090 be sufficient ( 24 GB ) ?

I would like to experiment by training the model on custom datasets, is it possible with a 4090 ? If not, what would be the hardware requirements ?

Thank you and have a nice day !

[randyjunior2008]

Same question, can I train 30B model with mutiple 4090 24GB (x2 ~ x3) insead of single professional GPU (such like A100) ?

[olliestanley]

I would like to experiment with Open Assistant on my local setup, i browsed the repository and issues for hardware requirements ( VRAM ).

What are the VRAM requirements for running the model ? Would a 4090 be sufficient ( 24 GB ) ?

There are many different models. In general, to run any models in 24GB would require significant quantisation. You may be able to run the 30B LLaMA models on 24GB GPU using 4-bit quantisation.

You could probably also run the 12B Pythia models on 24GB GPU using 8-bit quantisation.

Quantisation will somewhat affect model output quality though.

As for training it really depends on how you do it, there are many different training methods which enable memory-efficient training, but you would have to look into them.