Support for Face Landmark Detection

[DebuggingBee]

Hi All,

My uncle has ALS. At this moment he is still able to speak and type, but we don't know how long this will be possible.
In my research for solutions, I only found very expensive (Tobii Dynavox) solutions. And the consumer version of Tobii Eye Tracker 5 is not supported for Windows Eye Control, because the Tobii 5 is licensed for only games.

In my research to come up with a solution where my uncle would still be possible to communicate with his eyes, I found OptiKey (ofcourse), but I also found Face Landmark Detection: https://developers.google.com/mediapipe/solutions/vision/face_landmarker

As I'm a developer myself, I developed a first concept (web application), things I tested:

• I created a fixed grid of 4x3
• I am able to focus to those items in the grid and change the background color
• I am able to detect winking / closing eyes. Just for testing I change the background color when I close my eyes. This goes very well, maybe it would be an option to detect this as a mouse click, or if the user closes their eyes for 3 seconds, it means: close application, or go back or something
• next step I want to test is calibration. For now: If I face the camera straight, it works with my static settings. Next step would be to use brainjs (sorry, I'm a web developer) to calibrate. If this goes well, it should even work when lying on bed, or looking to the camera from the side, when only one eye is visible.

Okay, now my question. Would it be possible to tinker on this, having a look if this can be supported by OptiKey as well. To my humble opinion, this technology can be a great solution instead of Dynavox products.

Regards, Peter

[alexandre-mbm]

Hello, @DebuggingBee!

Your investigation is very interesting for some stages of ALS. However, you should prioritize methods that are less costly for the patient.

(I express myself badly in English, so I save words. I speak portuguese of Brazil natively. You may do asks in english...)

Tobii EyeX and Tobii 5 works on Optikey. Tobii 5 will have a "restarting" issue for patients who are closing in on themselves. Tobii EyeX requires USB 3.0.

My father is in a advanced stage of ALS. The "Face Landmark Detection" is a great discovery! I will study it. Thanks.

[DebuggingBee]

Yeah, so the Tobii Dynavox solutions are no option. I want it to be free or as cheap as possible.
Here is another example, you need to scroll down, and then you will see all parameters live: https://codepen.io/mediapipe-preview/pen/OJBVQJm

[alexandre-mbm]

Yeah, so the Tobii Dynavox solutions are no option. I want it to be free or as cheap as possible.

Cost is a complex variable.

A "Tobii for gaming" on Optikey would give confort to your uncle for relevant period.

[alexandre-mbm]

@DebuggingBee, perhaps it as 3 or 5 virtual switches for ACAT (scanning paradigm).

[NaturalDevCR]

I'd love to know more about this, specially if you're planning to make a web development.