Echo-like voice assistant for healthcare use cases

[winghavenrc]

Ok, I'm not a deep technical guy though I've done a decent amount of Python and Nodejs dev to build a healthcare specific voice assistant. I built an Alexa skill (till they killed that option - it was never a great platform for h/c), tried some other open source conversational AI platforms (Mycroft, Neon) but really really intrigued by what you're doing with Willow.
Here's my question (and excuse my ignorance) - with the Willow Inference Engine, what other components will I need to build a full Echo-like conversational AI voice assistant? Not really sure where to put WIE in the NLP, Dialog Management, SST/TTS terminology that I'm more familiar with from the other platforms? Will I need something like RASA on the backend to do the NLP/Dialog Management?

Thanks and hoping your guys are very successful!

[kristiankielhofner]

Isn't this a coincidence!

I have a background in heath tech and several years ago I was hired to consult on a project to build a PHI respecting "completely in our walls" voice assistant project by a large publicly traded health tech company you've definitely heard of. I don't know if you were around but voice interfaces got a lot of renewed attention in healthcare with COVID because of sanitation issues with touch interfaces, etc.

That project failed because I was trying to use the ESP32 at the time and the entire ecosystem just wasn't ready. Not to mention Raspberry Pi approaches which everyone should realize by now are hopeless in terms of delivering true commercial/Alexa/etc user experience. Now with Willow and the ESP32 S3 we target it's clearly well within reach in terms of the challenging physical/audio issues.

Due to my network and experience in the field targeting healthcare specifically is actually our first monetization strategy. Here's the thinking:

• Willow devices (of course)
• Willow Inference Server for the STT, TTS, and LLM (it supports LLaMA/Vicuna and other derivatives today) - GPU stuff
• I actually used Rasa for the previously mentioned project (but delivering clean audio in healthcare settings - hard walls, reflective surfaces, etc wasn't possible with ESP32 - with Willow it is. If you can't fundamentally get clean speech you're dead in the water)
• The Willow Application Server (WAS) - not announced

Basically the WAS would be the API frontend for integrations (which I'm sure you know are a nightmare in healthcare) that would coordinate any number of backend WIS instances depending on desired scale, redundancy, etc (obviously K8s or along those lines). I haven't thought it through completely but Rasa, etc could also live on WAS. It would also do device management, provisioning, etc of the Willow devices themselves.

[winghavenrc]

Yep - I was running an HIT software company during Covid which is part of the reason that I'm digging into this space now... That's really interesting and It makes total sense given the privacy requirements! I considered the notion of a laptop-based VA but the cost and maintainability just doesn't work...

You're open source now, I assume that you plan on having a licensing model for enterprise deployments (Rasa-like) then?

I'll work on getting up to speed with Willow once my Lite arrives.

[kristiankielhofner]

Willow and the Willow Inference Server will always be open source. Not sure about the Willow Application Server at this point because I haven't fully thought it through yet but leaning towards the client implementation being in the open source tree for the Willow device and WAS itself being commercial only. That said the client will be open and the spec will be available so I'm expecting tighter integration for home users with Home Assistant, openHAB, etc using native modules/components on those platforms that would essentially emulate WAS for the integration and seamless end-user experience with those platforms.

This isn't my first open source project where I've balanced the needs and interests of the open source users while offering commercialization for business users and I think this is a reasonable, fair, and balanced approach.

[enzo2]

@kristiankielhofner As a physician, I discovered Willow via the Home Assistant universe, however I am really looking forward to what you can build in healthcare. It would be interesting to start a high-tech solo practice where I roll my own chatbots, diarization + note-generating stack, and administrative assistant, around a common inference server. I think for physicians, having "Echo" is less important than automating some of the most time-consuming functions, like writing notes, triaging queries, office work, etc.

Know you need to cater to larger organizations, but it could also help solo physicians to recapture the golden days of medicine.

[kristiankielhofner]

My background is more geared towards large hospitals and healthcare systems but you have a good point. The issue (I would imagine) with supporting small practices is the support, integration, and sales overhead. In many ways it's easier to sell and support 1,000 devices in a single hospital than it is to support 1,000 individual smaller practices; this is why I've always gravitated towards this approach to market. It's one sale (with a long cycle) and large healthcare systems have substantial dedicated IT resources that can support integrations which I'm sure you know is the most challenging aspect (EMR/EHR systems, etc).

That's not to say this isn't something we would consider at some point; it's just not where we will start because I don't know the market as well.

[enzo2]

Certainly understand, I work in the hospital setting mainly. Looking forward to your inevitable success in that area. But I'll keep tabs on your open source side of things and what I might be able to do with Willow. (Granted, I'm in the <1% of doctors who don't need tech support)