Is there a way to use two GPUs?

[rookiemann]

Hello, first off I am really impressed with this and would like to say great work.

I was curious if there would be a way to use two GPUs? I have two 8GB Nividas (GTX 1080 and RTX 2080) and I would like to use the larger models more efficiently. For instance when I try the Facebook large music gen I get out of memory CUDA errors when I go above 8 seconds.

Amazing stuff, thank you for this.

[gitmylo]

Currently it only uses the first detected gpu, models such as musicgen don't currently support multi gpu inference as far as i know though, correct me if i'm wrong.

[rookiemann]

I use the webgui for Stable Diffusion a lot and it can't do it, so I'm thinking there probably isn't a way here neither, just wanted to ask to make sure. What happens is I'll get an out-of-memory CUDA error and I'll only be short barely half a gig of memory or something and I really wish I could unload some of that processing to another GPU.

[gitmylo]

It's technically possible for me to add settings to send certain models to certain gpus, so you can for example have musicgen and bark loaded on different gpus. I'd have to rewrite a bunch of cuda checks to have a different one instead though.

[rookiemann]

This repo is pretty solid and I've been trying a lot of different models for different things experimenting and I'm thoroughly impressed as-is with it. You don't have to do a bunch of stuff just for me :) but maybe if you were to thinking of it maybe it can be done in half-precision? I really don't know just spit balling here.

[gitmylo]

I'm not sure if huggingface transformers allows easy half precision, I'll keep it in mind though.

[rookiemann]

Right on, thank you very much for your time this Sunday morning I do appreciate it :).

[rookiemann]

Just an update, I've installed this repo on a remote server with 64GB vram and wow I am just blown away. This is really powerful and impressive.

One thing I'd like to suggest is batch processing with random seed each time with music generators, so that many samples can made at once and mixed as needed.

I'm making techno music and it's wild, I can ask it to do intros or drops, BPM, or specify breaks and bridges and it delivers. I'm sure I can finally be that DJ I always wanted to be lol. This is a good repo very pleased with it.

[gitmylo]

That would be a cool thing to add, although i'm not sure if i could test it.

[rookiemann]

I would test it for you :), but there's really not much to test because all it would need to do is just keep looping the commands, but with a different random seed each time. My Python knowledge is so-so, maybe I'm overstepping here.

Since I've put this up on the remote server the generations are so fast now, I'm noticing they finish before I notice, and I have to keep hitting generate for another, which is ok I'm not complaining because I can do that. But if there were batch processing, there would be already 50 made by the time I remembered to check it again lol.

[gitmylo]

Yeah but 64gb would allow batching, that would be the special part. Doing sequential generations isn't that special, but with batches you can generate multiple generations at once.

Although i would again run into a previous issue, gradio audio output elements only support one file to be outputted, not multiple.

[rookiemann]

Ah ok understood, thank you. Just an FYI, the only file I'm interested in is the .wav file. All those .mp4 and .mvk I don't need.