SSL w/o torchaudio dependency

[wanchichen]

What?

This PR allows for HuBERT pre-training w/o using torchaudio, allowing for more customization and use of different ESPnet components. It also introduces some tricks to better support large-scale training.

Features:

• Flash Attention (by @pyf98)
• Activation checkpointing in E-Branchformer
• Convolutional Feature Extractor as frontend
• Convolutional positional embeddings
• Efficient batch sampler for large-scale
• Efficient multi-shard iterator for large-scale
• WavLM noise augmentation
• More efficient distributed training
• Recommended tunable DDP args for multi-node training
• Cross-Entropy loss for HuBERT SSL
• Intermediate losses for HuBERT SSL
• HuBERT SSL using filterbank input features
• More detailed GPU memory reporting

Supports only HuBERT for Transformer and E-Branchformer so far.

To do:

• Fbank training configs
• Conformer, branchformer encoder
• check in hubert.sh which implementation to use

[mergify]

This pull request is now in conflict :(

[sw005320]

@wanchichen, can you restart this PR?

[mergify]

This pull request is now in conflict :(

[sw005320]

@wanchichen, let’s finish this PR.
There are a lot of conflicts now. So, please resolve them.

@simpleoier, please also review this PR.

[wanchichen]

I performed several experiments (ASR, SSL) and found that this was useless. But we may need to experiment with other batch samplers to make sure

[wanchichen]

This is the recommended setting by PyTorch now, it is both faster and more memory efficient. However, it may also slightly affect the learning curve

[simpleoier]

Thanks! I made some comments.
One question I have is about the espnet2/ssl/mask. What is the benefit of a new mask module? If it is only used in HuBERT models during pre-training, a single function in the hubert model would be enough.