End-to-end Speech Translation

This repository is the official implementation of the following papers:

• "Listen, Understand and Translate": Triple Supervision Decouples End-to-end Speech-to-text Translation

  Qianqian Dong, Rong Ye, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, Lei Li. AAAI 2021.

  [Project Page]

• Consecutive Decoding for Speech-to-text Translation

  Qianqian Dong, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, Lei Li. AAAI 2021.

  [Project Page]

Requirements

• Python 3
• Tensorflow 1.15
• Required packages are listed here.

To install requirements:

pip install -r requirements.txt

Data Preprocessing

LUT

python3 -m st.tools.dataset configs_template/lut.yaml

COSTT

python3 -m st.tools.dataset configs_template/costt.yaml

Train

LUT

python3 -m st.bin.run_lut -m train -c configs_template/lut.yaml

COSTT

python3 -m st.bin.run_costt -m train -c configs_template/costt.yaml

Decode

LUT

python3 -m st.bin.run_lut -m infer -c configs_template/lut.yaml

COSTT

python3 -m st.bin.run_costt -m infer -c configs_template/costt.yaml

Citations

Please consider citing our papers in your publications if the project helps your research. BibTeX reference is as follows.

@inproceedings{dong2021listen,
  title={Listen, Understand and Translate: Triple Supervision Decouples End-to-end Speech-to-text Translation},
  author={Qianqian Dong, Rong Ye, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, Lei Li},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2021}
}

@inproceedings{dong2021consecutive,
  title={Consecutive Decoding for Speech-to-text Translation},
  author={Qianqian Dong, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, Lei Li},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2021}
}