When you lose your vision, you lose contact with things.
When you lose your hearing, you lose contact with people.
Helen Keler

In South Korea, changes to the health insurance criteria in 2009 expanded cochlear implant surgery and enabled more kids with hearing impairments to study in inclusive education environments.[1] However, despite many receiving inclusive education, there aren't enough services for kids with hearing impairments in inclusive education classrooms in South Korea. These kids mostly communicate with kids without disabilities through speech.

This environment causes a lot of discomfort for children with hearing impairments. According to Ms. A, a person who has a hearing impairment, one of the difficulties in regular school was not being able to check her own voice's volume and content, which made her continuously ask friends to check them. Therefore, we need a service for kids with hearing impairments who need to engage in everyday communication in inclusive educational settings.

Team Allways build "All Ears", an WearOS app that provides real-time Voice-Noise Ratio Visualization, Live Captioning, and Past Captioning Rewatch for kids with hearing impairment. We aim to build an additional ear through visualization of speech and help kids with disabilities build camaraderie with their friends. With "All Ears" we can achieve 3 UN SDGs goals—the quality of education, promote peace, and reduce inequalities.

To achieve Voice Ratio Visualization, we utilized TensorFlow Lite's AudioClassifier model, an on-device AI for edge devices. We also implemented Live Captioning using GCP's STT API. For Past Captioning Rewatch, we created a Spring Boot (Java 17) backend server that uses GCP Cloud SQL (MySQL 8.0 engine) and hosted it on a GCP VM instance.

These features are visualized through an app built on WearOS (Kotlin 1.7 & Java 8), making it possible for users to access it in everyday situations at school.

This project has two repositories: All-Ears_Android for WearOS client application and All-Ears_Server for SpringBoot backend server. The guidance on how to run the code is in each repository's GitHub README.md file. If you want to run the code, please follow the instructions of each repositories.

v1.0.0 release

• Added Voice-Noise Ratio Feature(Tenserflow lite)
• Added Speech-to-Text Featrue(GCP STT API)
• Added Past Captioning Rewatch Feature(GCP Cloud SQL & VM instance)

Android	Android	Backend	Design
노원희/Wonhui Roh	정예지/Yeji Jeong	전혜승/Haeseung Jeon	김수연/Suyeon Kim

[1]Cho, Soo Jin, Mi Sook Lee, and Yu Ri Lee. "Analysis of the Current Status and Characteristics of the Hearing Impaired Using Data from the Actual Survey for the Disabled." Audiology and Speech Research 18.1 (2022): 9-19.