NASTAR: NOISE ADAPTIVE SPEECH ENHANCEMENT WITH TARGET-CONDITIONALRESAMPLING

This is the official implementation of our paper. Our work applies resampling techniques on existing source datasets to the noise adaptive speech enhancement task in a one-shot scenario. The resampling scheme can achieve higher scores with less training data.

Current Information

This project is partially public. The source code of SE has been released. We will open the implementation of our resampling method if our paper gets accepted. Those checkpoints and logs under all the noise adaptation settings and the testing data are provided. The pseudo-noise and relevant-cohort demonstration for NASTAR can be found here.

Contents

• Installation
• Steps and Usages
• Contact

Installation

Note that our environment is in Python 3.7.10. To run the experiment of NASTAR, you can clone the repository and install it by using the pip tool:

git clone https://github.com/ChangLee0903/NASTAR
cd NASTAR
# Install all the necessary packages
pip install -r requirements.txt

Steps and Usages

1. Data Preprocess:

   You can produce your own pair data by using the Corruptor class in data.py and set the data path in config/config.yaml. Note that all the training noisy utterances are generated online. Our source noise dataset is provided by DNS-Challenge. The target_data directory contains the noise signals test.wav used for testing and the results of pseudo-noise pesudo.wav / cohort-set cohort.txt. Check dataset in config/config.yaml:

   dataset:
    train:
      speech: ../speech_data/LibriSpeech/train-clean-360
      noise: ../noise_data/DNS_noise
    dev:
      speech: ../speech_data/LibriSpeech/dev-clean
    test:
      data: ../NASTAR_VCB_test_data
    ...
    

   Since we only test adapted model on one specifice noise type, there is no need to set the noise dataset path in dev. Instead, we use --eval_noise to assign the noise signal for evaluation. To avoid randomness as testing, the testing data have been mixed and put in NASTAR_VCB_test_data. All the noisy and clean utterances were saved as npy files.

2. Training the adapted SE model for each target noise condition:

   First, download the checkpoint of pretrained model in the PTN directory, and make sure the argument --ae_ckpt has been set correctly. All the recipes of different settings are recorded in our repository so that you can just run the script. Note that you can set --device to change the identity of the used CUDA device.

   bash train_NASTAR.sh
   

3. Evaluation:

   Make sure all of your models have been trained and stored in the path of --ckptdir. All the results will be stored in dictionaries and saved as pth files.

   • Check ckpt directory
   • Run:

   python main.py --task test --n_jobs 16

Contact

Any bug report or improvement suggestion will be appreciated!

e-mail: r08922a28@csie.ntu.edu.tw