Fast Conformer Fine-Tune

[SuperReem]

I'd like to fine-tune the pretrained NeMo model 'Conformer-CTC-L_spe128_ar-AR_3.0.nemo' using a multilingual fast conformer configuration. Which configuration file (.yaml) should I use for this purpose? And could you suggest how to incorporate this configuration file into the 'speech_to_text_finetune.py' script?

[nithinraok]

Conformer and fastconformer have different subsampling modules, you can copy the conformer modules without subsampling while finetuning from a pretrained module.

https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/asr_finetune/speech_to_text_finetune.yaml
you can add include and exclude modules to init_from_nemo_model as shown here

NeMo/nemo/core/classes/modelPT.py

Line 1194 in 8349651

init_from_nemo_model:

[SuperReem]

@nithinraok
I don't have a pretrained fastconformer checkpoint. How can I modify the initialization process to exclude the subsampling module from the conformer? In the line exclude["decoder"], what should I replace "decoder" with to exclude the subsampling module?

[nithinraok]

Similar to here:

NeMo/examples/speaker_tasks/recognition/conf/titanet-finetune.yaml

Line 4 in 32b9ae6

init_from_pretrained_model:

in exclude section add encoder.pre_encode

[SuperReem]

After excluding encoder.pre_encode:

Error executing job with overrides: ['exp_manager.exp_dir=./outputs/output_5', '+model.tokenizer.update_tokenizer=true', 'model.tokenizer.langs.en.dir=./en_tokenizer', 'model.tokenizer.langs.ar.dir=./tokenizer', '+init_from_nemo_model.model0.path=Conformer-CTC-L_spe128_ar-AR_3.0.nemo', '+init_from_nemo_model.model0.exclude=[encoder.pre_encode]']
Traceback (most recent call last):
File "/home/ma-user/work/multilingual_fast_conformer/./speech_to_text_ctc_bpe.py", line 85, in main
asr_model.maybe_init_from_pretrained_checkpoint(cfg)
File "/usr/local/lib/python3.10/dist-packages/lightning_utilities/core/rank_zero.py", line 32, in wrapped_fn
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/nemo/core/classes/modelPT.py", line 1248, in maybe_init_from_pretrained_checkpoint
self.load_part_of_state_dict(
File "/usr/local/lib/python3.10/dist-packages/nemo/core/classes/modelPT.py", line 1148, in load_part_of_state_dict
self.load_state_dict(dict_to_load, strict=False)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 2091, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for EncDecCTCModelBPE:
size mismatch for decoder.decoder_layers.0.weight: copying a param with shape torch.Size([129, 512, 1]) from checkpoint, the shape in current model is torch.Size([257, 512, 1]).
size mismatch for decoder.decoder_layers.0.bias: copying a param with shape torch.Size([129]) from checkpoint, the shape in current model is torch.Size([257]).

[nithinraok]

[encoder.pre_encode]'] I see an extra ] in the submodule name

[nithinraok]

You should also exclude decoder as shown from the error