Detect solfege hand signs using machine learning (live demo)
a project by Ryan Kemmer and Josh Stovall
We have developed a system that allows the user to 'play' solfege hand signs as a MIDI instrument. This technology could have practical applications both in music education and creation.
Solfege hand signs are often used in school choirs to teach music theory concepts. Since vocalists cannot actually see the notes on their instrument, hand signs have been developed as a teaching aid.
This system allows students to play the hand signs as an instrument -- recieving auditory feedback. Previously, a student learning solfege would have to sing the note or play another instrument to 'hear' their hand signs.
This project uses TensorFlow, MediaPipe, and JZZ.js.
cd solfege
php -S 0.0.0.0:1234
go to http://localhost:1234
- make a hand sign
- click the button to create training data
- it will wait 1 second, start recording data for 30 seconds, then save a file with the data
| hand sign | name | pitch class (distance from C) |
equivalent note |
|---|---|---|---|
 |
do | 0 | C |
 |
di | 1 | C# |
 |
ra | 1 | Db |
 |
re | 2 | D |
 |
ri | 3 | D# |
 |
me | 3 | Eb |
 |
mi | 4 | E |
 |
fa | 5 | F |
 |
fi | 6 | F# |
 |
se | 6 | Gb |
 |
so | 7 | G |
 |
si | 8 | G# |
 |
le | 8 | Ab |
 |
la | 9 | A |
 |
li | 10 | A# |
 |
te | 10 | Bb |
 |
ti | 11 | B |
Any contributions you make are greatly appreciated. Any bugs and change requests are to be reported on the issues tab. If you don't like coding, you can contribute by becoming a sponsor.
Please write to contact@instrument.bible or visit instrument.bible.
