Tone.me is a platform built at MIT aimed at improving Mandarin pronunciation.
Developed by Chris Ge, Daria Kryvosheieva, Anshul Gupta, Riddhi Bhagwat, & Katherine Guo
Table of Contents
Our three-step process accepts a Chinese voice recording,determines the tones in the recording, and returns feedback on the correctness of the tones.
- Audio Input. Record an audio of yourself pronouncing a character, word, phrase or sentence. Specify what you want to say.
- Analysis. Our software predicts tones from the audio and compares to the intended ones.
- Feedback. The software returns whether or not your pronunciation was right, and displays the audio of your pronunciation of each syllable compared to an authentic pronunciation
We used the Data-AISHELL3 Mandarin Speech Corpus as raw data split it into audio segments of individual syllables using Whisper and stable-ts, filtered the syllables using Pydub, and fine-tuned the wave2vec2 transformer model on the resulting audio segments. Here's a diagram of our model pipeline:
The code used to train the model is included in the model folder, but it wasn't written for public distribution and is only included for reference. Our model is publicly available on Hugging Face.
Some of the main frameworks and libraries that we used in our project
Our service is no longer hosted on https://tone-me.onrender.com because of cost reasons. However, here is how you can set up Tone-Me for use on your local computer.
We recommend that you install the following packages within a virtual environment.
- npm
npm install
- pip
pip install -r requirements.txt
Then, you can run python3 app.py in one shell, and npm run dev in another shell, and access your website at localhost:3000.
Distributed under the MIT License. See License.txt for more information.
Chris Ge - cge7@mit.edu | Riddhi Bhagwat - riddhib@mit.edu | Anshul Gupta – anshulg@mit.edu
This project would not have been possible without the support from AIM Labs. We're especially grateful to Sagnik Anupam for all of his advice and the opportunity we had to present during Demo Day.