Google colab ?

[CarloCattano]

Could anyone train RAVE on a google colab ?
Im trying many different approaches and get stuck with division by zero

File "/usr/local/lib/python3.7/dist-_reset_eval_dataloader
    min_pct = 1.0 / len(dataloader)
ZeroDivisionError: float division by zero 

when running train_rave.py

Would be neat to use colab for it when not having a powerful GPU at home

[CarloCattano]

full error output :

21.7 M    Trainable params
0         Non-trainable params
21.7 M    Total params
86.972    Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/data.py:128: UserWarning: Total length of `DataLoader` across ranks is zero. Please make sure this was your intention.
  f"Total length of `{dataloader.__class__.__name__}` across ranks is zero."
Traceback (most recent call last):
  File "train_rave.py", line 160, in <module>
    trainer.fit(model, train, val)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 769, in fit
    self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run
    results = self._run_stage()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1321, in _run_stage
    return self._run_train()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1343, in _run_train
    self._run_sanity_check()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1404, in _run_sanity_check
    val_loop._reload_evaluation_dataloaders()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 239, in _reload_evaluation_dataloaders
    self.trainer.reset_val_dataloader()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1949, in reset_val_dataloader
    RunningStage.VALIDATING, model=pl_module
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 424, in _reset_eval_dataloader
    min_pct = 1.0 / len(dataloader)
ZeroDivisionError: float division by zero

[raniczny]

Hey @CarloCattano, I tried and it worked for me around a month ago (latest commit was 5eb5a1b). Not sure what causes the error you see.

#@title 1.1 Check GPU time

import subprocess
!nvidia-smi

#@title 1.2 Anti disconnect from Colab

import IPython
js_code = '''
function ClickConnect(){
console.log("Working");
document.querySelector("colab-toolbar-button#connect").click()
}
setInterval(ClickConnect,60000)
'''
display(IPython.display.Javascript(js_code))

#@title 1.3 Install dependencies

#@markdown - torchmetrics 0.8.0 doesn't work, so need 0.7.3
#@markdown - original torch version from the requirements.txt doesn't seem to work, so commented out

!pip install torchmetrics==0.7.3
!pip install tqdm==4.62.3
!pip install effortless-config==0.7.0
!pip install einops==0.4.0
!pip install librosa==0.8.1
!pip install matplotlib==3.5.1
!pip install numpy==1.20.3
!pip install pytorch_lightning==1.4.9
!pip install scikit_learn==1.0.2
!pip install scipy==1.7.3
!pip install soundfile==0.10.3.post1
!pip install termcolor==1.1.0
# !pip install torch==1.11.0
!pip install tensorboard==2.8.0
!pip install GPUtil==1.4.0
!pip install git+https://github.com/caillonantoine/cached_conv.git#egg=cached_conv
!pip install git+https://github.com/caillonantoine/UDLS.git#egg=udls

#@title 1.4 Clone RAVE repo and begin config
import sys

!git clone https://github.com/acids-ircam/RAVE.git
sys.path.append("./RAVE")

# maybe git checkout to that commit above if the latest master doesn't work

!python RAVE/cli_helper.py

After this you just run commands from the helper output. In my case files were served from Google drive

!python RAVE/train_rave.py --name <MODEL_NAME> --wav drive/MyDrive/<PATH_TO_WAV> --preprocessed <DIR> 

# ... etc.

Hope that helps.

[raniczny]

Btw, current master (16ea4a9) seems to works fine, no need to modify dependencies versions anymore. Not yet sure if something's going to fail with later stages (export, prior), but for now training runs fine.

[HenningSchaar]

Thank you so much! After almost giving up with Ubuntu on arm64 this is the farthest I have gotten in this process. However I am currently stumbling over the process of exporting the latent space...

After training RAVE and killing that process i tried exporting the latent space with following command (provided by cli_helper.py):

!cd RAVE && python export_rave.py --run runs/emotionalspeech_v006/rave --cached false --name emotionalspeech_v006

[230.60] exporting model
[1674.43] loading model from checkpoint
Traceback (most recent call last):
  File "export_rave.py", line 206, in <module>
    RUN = search_for_run(args.RUN)
  File "/content/RAVE/rave/core.py", line 192, in search_for_run
    run_path = list(filter(lambda e: "last" in e, run_path))[-1]
IndexError: list index out of range

Am I missing something? Has someone else gotten this to work? Sorry if it is obvious, I am a Sounddesigner trying to wrap my head around all this!

[raniczny]

Hey @HenningSchaar, it looks like there's nothing to export, make sure you have checkpoints.

[HenningSchaar]

Thank you! I was just too impatient. After waiting for Checkpoints to be created, everything works.

[asantos-6]

Hi! So I trained a fairly small model , v2_small config, and I was trying to export it, but apparently, my runs don't have any checkpoints... why is that? Is there a minimum number of steps for the first checkpoint? I ran the model for just 1000 steps

[CarloCattano]

Still no luck , I get the same error as before , but with a new mention to "Missing logger folder"

audio_00000_00000.wav: 100% 12/12 [00:00<00:00, 22.56it/s]
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:490: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  cpuset_checked))
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Missing logger folder: runs/amentest/rave
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name          | Type                | Params
------------------------------------------------------
0 | pqmf          | CachedPQMF          | 16.7 K
1 | loudness      | Loudness            | 0     
2 | encoder       | Encoder             | 1.4 M 
3 | decoder       | Generator           | 3.4 M 
4 | discriminator | StackDiscriminators | 16.9 M
------------------------------------------------------
21.7 M    Trainable params
0         Non-trainable params
21.7 M    Total params
86.972    Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/data.py:128: UserWarning: Total length of `DataLoader` across ranks is zero. Please make sure this was your intention.
  f"Total length of `{dataloader.__class__.__name__}` across ranks is zero."
Traceback (most recent call last):
  File "train_rave.py", line 159, in <module>
    trainer.fit(model, train, val, ckpt_path=search_for_run(args.CKPT))
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 769, in fit
    self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run
    results = self._run_stage()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1321, in _run_stage
    return self._run_train()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1343, in _run_train
    self._run_sanity_check()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1404, in _run_sanity_check
    val_loop._reload_evaluation_dataloaders()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 239, in _reload_evaluation_dataloaders
    self.trainer.reset_val_dataloader()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1949, in reset_val_dataloader
    RunningStage.VALIDATING, model=pl_module
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 424, in _reset_eval_dataloader
    min_pct = 1.0 / len(dataloader)
ZeroDivisionError: float division by zero 

[raniczny]

I also get the same warning about the folder, but it doesn't affect anything in my case.

Are you using Colab free or Pro?
I see This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, in Colab Pro it also shows this warning, but suggests 4 worker processes. Not sure if it has anything to do with the error you get, though.

And another one -- could it be a problem with your dataset? From what I see here len(dataloader) returns a number of batches, and in your case it's 0. For me it works with a rather small dataset, but maybe the nature of sound could also affect that?

$ duration
Found 4 .wav files
total duration: 0h 25m 31s

[CarloCattano]

OMG ! It was just too small of a dataset ! such a dumb issue ...
I was using just a few short samples to test the enviroment out before doing any actual training and well... Now I know that
Thanks for the hints

[raniczny]

Ah, great, glad it's solved!

[CarloCattano]

I also get the same warning about the folder, but it doesn't affect anything in my case.

Are you using Colab free or Pro? I see This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, in Colab Pro it also shows this warning, but suggests 4 worker processes. Not sure if it has anything to do with the error you get, though.

And another one -- could it be a problem with your dataset? From what I see here len(dataloader) returns a number of batches, and in your case it's 0. For me it works with a rather small dataset, but maybe the nature of sound could also affect that?

Thanks to this comment of @raniczny I just realized it was working all the time , just my dataset was too small . I intended to test the dev enviroment first with dummy audios that were very short and not so many (facepalm) thats why the dataset was so tiny that was triggering a weird "divided by zero"

[caillonantoine]

Hello :) I added a minimum size to the validation set in e26bdfd : now even if your dataset is small, it should work!

[CarloCattano]

Awesome, now I got collab pro and trying to train a 40min long model of didgeridoo optimized for the raspberry pi. Will try with smaller datasets once this one finish, but I understand that the longer the better, I was just trying out with too less samples to check that I did everything correctly

[CarloCattano]

For future readers , here are the modifications to use in colab including tensorboard setup

# check that the gpu is assigned
import subprocess
!nvidia-smi

-----------------------------------

# mount google drive
from google.colab import drive
drive.mount('/content/drive')

------------------------------------

# Clone repo and install requirements
%cd /content/drive/MyDrive
!git clone https://github.com/acids-ircam/RAVE.git
%cd /content/drive/MyDrive/RAVE
!pip install -r requirements.txt

------------------------------------

#Resample and augment samples folder
!resample --input /content/drive/MyDrive/samplesFolder --augment --sr 48000 --output /content/drive/MyDrive/Augmented

------------------------------------

!python cli_helper.py

------------------------------------

#setup colab tensorboard
%load_ext tensorboard
%tensorboard --logdir . --bind_all 

------------------------------------
# train the model
%cd /content/drive/MyDrive/RAVE
!python train_rave.py -c small --name modelName --wav /content/drive/MyDrive/Augmented/out_48000 --preprocessed /content/drive/MyDrive/temp/modelName/rave --capacity=4 --ckpt /content/drive/MyDrive/RAVE/runs/modelName/rave/version_x

------------------------------------

!python export_rave.py --run runs/modelName/rave --cached true --name modelName_rt

[branlsnyder]

I'm having trouble parsing out how to resume training from my last checkpoint (I trained ~700k steps, timed out, and now want to continue from where I left off). When I resume training, a new version is created, but it seems to start from 0 steps and Epoch 0 again. Am I missing something?

My input (in google colab pro):

!python train_rave.py -c small --name young-hi_1-24_haegeum-training_01 --wav /content/drive/MyDrive/RAVE/in-files/young-hi_ji_1-24_preprocessed --preprocessed /content/drive/MyDrive/RAVE/temp_01/young-hi_1-24_haegeum
-training_01/rave --capacity=4 --ckpt /content/drive/MyDrive/RAVE/runs/modelName/rave/version_0

And a screenshot of my tensorflow

EDIT: thanks Carlo for facilitating this super helpful discussion thread about the colab!

[CarloCattano]

You must provide the absolute path of the desired checkpoint content/drive/MyDrive/RAVE/runs/modelName/rave/version_0/last.ckpt or similar

[branlsnyder]

ah ok. Thats pretty obvious, I suppose. Thanks

[jeanbrazeau]

I'm curious how many steps you can actually log with Google Collab even with a small dataset considering the resource constraint of 12 hours... are people hitting <1-million steps?

[olaviinha]

Is there some trick to continue previous training after an interruption? Don't have the error log right now, but I think it failed for me the first time I tried. version_n+1 is always a continuation of version_n or how does it work?

Also offtopic, but bulgarianvoices, I like where you're going!

[CarloCattano]

yes you are able to "resume" a training by using --ckpt /path/last.ckpt of a given version. I still get some warnings about it from time to time depending on how colab decides to kick me out after some time , but I'm hoping it will work without side effects.

I find Bulgarian voices so mesmerizing , I though would be a good data to practice on till I get some level of confidence to feed it with my own material , since there is long recordings available and was curious to try :-)

[olaviinha]

Oh yeah, that worked, thanks! I think I tried to define only directory path (not file path) the first time, probably.

Is the training slowing down for you as it progresses? First run reached 1M steps, but seems like it slowed down considerably when I continued it. In the second (continuation) run, only +230k steps during ~16 hrs so far, whereas the first training session reached 700k steps in the same time. Same GPU both times. Wondering if this is normal. Are you having similar experiences?

[caillonantoine]

this is normal, starting from 1M step the second stage of the training kicks in, which is considerably heavier! 🚂

[smithee77]

hi, colab pro + here. I'm working on a really small dataset (10 audios of 4 secs). How long should I train RAVE and PRIOR to get reasonably good results? is it 1.000.000 epochs compulsory??
many thanks

[CarloCattano]

I do see this warning , using Colab Pro but Im not aware of the limitations or workarounds

 UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.

[CarloCattano]

Im tempted to change it to 2 in here , but not sure about the consequences

RAVE/train_rave.py

Line 117 in e26bdfd

train = DataLoader(train, args.BATCH, True, drop_last=True, num_workers=8)

@raniczny , Im using regular pro now and still get the same suggested amount . Have you ever tried changing the value of num_workers ?
Maybe colab pro+ is the one giving more workers access ?

[raniczny]

@CarloCattano I have not tried changing that yet.

I'm using the regular Pro. From my recent observations you get more logical cpus on the High-RAM environment (Change runtime type). You can check with cat /proc/cpuinfo. On the Standard runtime I've been getting a 1-core CPU (2 threads in total), on High-RAM -- 2-core CPU (4 threads in total).

[CarloCattano]

Cristal clear ! I will try with high RAM mode and see . I forked RAVE and tried to change the batches and num_workers and it didnt go so well

[CarloCattano]

You were right, with high memory seems much better and the warning just verifies 4 available workers and 8 called

[CarloCattano]

Also this is weird on resuming from a last-v1.ckpt :

model_checkpoint.py:343: UserWarning: The dirpath has changed from 'runs/bulgarianvoices/rave/version_0/checkpoints' to 'runs/bulgarianvoices/rave/version_1/checkpoints', therefore `best_model_score`, `kth_best_model_path`, `kth_value`, `last_model_path` and `best_k_models` won't be reloaded. Only `best_model_path` will be reloaded.
  f"The dirpath has changed from {dirpath_from_ckpt!r} to {self.dirpath!r},

[CarloCattano]

Shared some newbie questions on pyTorch forums :
https://discuss.pytorch.org/t/newbie-questions-torch-on-google-colab/151656
whish me luck

[CarloCattano]

I quitted my colab subscription. Was not happy how they disconnect you and the additional hassles with storage etc. For now I will try to run it on a humble machine or wait for other services. Suggestions welcome alternatives to Google colab

[branlsnyder]

I've considered AWS virtual machines, as they charge by the data-usage and not the time. But I haven't calculated exactly what that cost would be.

[olaviinha]

What is additional hassles with storages?

If you find anything even close to Colab subscriptions price-quality ratio, please ping me too. I don't have any problems with Colab but I'm definitely interested.

[CarloCattano]

My drive was getting full by the 2 or 3 millionth step , then of course got a message asking me kindly if I wanted to purchase more storage ...

[caillonantoine]

You can pass --val-every 100000 to decrease the number of validations, thus reducing the memory usage :)

[branlsnyder]

What is additional hassles with storages?

If you find anything even close to Colab subscriptions price-quality ratio, please ping me too. I don't have any problems with Colab but I'm definitely interested.

After a delay, I've started a training with an AWS Sagemaker Jupyter Notebook instance. So far it is nice (background execution, etc). Haven't reached any results yet, but price-wise it is a bit more than Colab Pro. the cheapest GPU-tier is 0.98 USD an hour. Their free tier instances don't have a GPU and are not worth it compared to Colab Pro. Hard to say what 2 million steps will take time-wise, as the second stage takes longer to train than the first, but it looks like it might be ca 60 hours. But so far, the background execution and ability to scale up makes it seem like a good solution for a big project (multiple models, a budget, etc), but for exploring and experimenting, Colab Pro might be a bit better.