Creating a new pipeline for all commands that result in "Sorry I couldn't understand that"

[jazzmonger]

So in thinking through this new Willow eco system now that I generally understand the basics of WIS, ha Voice assistants, etc, many (over 50%) of the voice commands I use every day aren't implemented in HA via its Assistant. As a stop gap, is there a way to intercept (or know programmatically) when a command sent to HA via WIS fails with "Sorry I couldn't understand that" And then perform another action?

Use cases (there are dozens, especially for integrations that only Alexa supports)
Alexa play x on y (Pandora and Apple music especially, not supported on Music Assistant with HA Supervised installs)
Alexa set a timer called t on z
Alexa what's the name of that song
Alexa add carrots to my Walmart list. (AnyList integration)
Alexa how many ounces in 1 cup
Alexa what is 2050 divided by pie
Alexa what is a recipe to substitute buttermilk

And on and on an on and on. Dozens (hundreds) more...

On such a failure (better yet, a new Trigger in an automation), I would send the same STT result from WIS on to Amazon's Alexa using this service:

This would allow the commands that HA understands to execute within the HA system, and pass the rest of them that fail on to Alexa to do something with. I know the whole point is to replace these nasty, cloud based spy boxes, but we just aren't there yet.

I would change Alexas wake word to Echo, and shim the Box3/WIS solution squarely between HA and Amazon's Alexa and slowly send fewer and fewer requests on to Amazon as time goes by and HA matures in the Voice arena.

Using the Alexa custom command service I've noticed that since you bypass the Amazon wake word and Amazons spotty STT, the result is pretty snappy.

How might I do this?

Jeff

[kovrom]

If you are willing to use WAC you can do it now.
https://github.com/kovrom/willow-autocorrect

[jazzmonger]

Absolutely brilliant! I'm 1/2 way there installing WAC, but aging parents demanded I abandon that effort, for a few days anyway. Looks like a perfect solution!

[jazzmonger]

Wait a sec... is this in your fork or the main branch? Are you romanKO on Discord?

Jeff

[kovrom]

It's in my fork. And yes, that would be me.

[jazzmonger]

Ok, I just figured that out. This is like a game changer. Now I want to drive the 10 hours to get home so I can play with this, but... I'll have to wait until after Xmas! How do your changes get folded back into the main branch or do they? I assume this is something everyone with Alexa or Google will absolutely want... at least initially.

so... does this work well? What kind of delay is there to ultimately fire off the automation after it's triggered?

[kovrom]

I see it as a stopgap, until HA intents are matured and local AI is more affordable hardware-wise. Trigger time on WAC side about 80ms on average. And then it is however long it takes your amazon to respond, I would say maybe 4 sec total.
Also, Merry Xmas :)

[jazzmonger]

I see it as a stopgap, until HA intents are matured

EXACTLY!! Which is the same as how I envisioned it. It's sort of the best of both worlds. As time goes on, fewer and fewer phrases would get forwarded to the Evil Empires and processed locally.

so, see if they can include this in the main branch so we can keep up with their updated releases!

you do realize that the combination of all of this completely leapfrogs ALL of the HA voice stuff by like a year... or more.