Adding gesture to wake word detection - is there a better way?

[mjhewitt1]

I am looking to add a gesture to the wake word detection process. The gesture recognition happens on a smart watch and when a gesture is recognised, a 1 is sent over esp now to the ESP-BOX.

To make it so that the ESP-BOX only wakes up if the wake word is detected AND a gesture is recognised, I have edited the audio_recorder_update_state() function in willow/deps/esp-adf/components/audio_recorder/audio_recorder.c so that the state only changes from IDLE if the wakeword is detected && gest_now == 1. I have added another function to audio_recorder.c so that the value of gest_now is updated when data is received from the watch device. The changes are outlined below.

It currently works, however I am wondering if you know of a better way to do this whereby I do not need to alter the esp-adf library? Thanks :)

---

willow/deps/esp-adf/components/audio_recorder/audio_recorder.c - additions:

typedef struct struct_message {
uint8_t wake;
} struct_message;
struct_message gesture;

uint8_t gest_now;

…

void onDataReceived(const uint8_t *macaddr, const uint8_t *data, int dataLen) {
memcpy(&gesture, data, sizeof(gesture));
gest_now = gesture.wake;
ESP_LOGE(TAG, "GEST NOW %d", gest_now);
}

…

static void audio_recorder_update_state(audio_recorder_t *recorder, int event)
{
switch (recorder->state) {
case RECORDER_ST_IDLE: {
if (event == RECORDER_EVENT_WWE_DECT && gest_now == 1) {
…….
}
break;
}
…………..
}

willow/main/main.c - additions

void app_main(void)
{
….
if (esp_now_init() != ESP_OK) {
ESP_LOGI(TAG, "ESP_NOW_INIT FAILED");
return;
}
….
}

[kristiankielhofner]

Thanks, this is very interesting!

Generally speaking we try to avoid patching esp-adf wherever possible. @stintel is currently on vacation but we'll take a look at this when he gets back; we may be able to come up with a way to implement this in a more generic fashion (and potentially without patching esp-adf).

We're looking at the ability to programatically trigger wake start/stop and vad start/stop with esp-adf which would likely enable functionality like this without patching esp-adf for each use case.

Stay tuned!

[mjhewitt1]

Thanks for the response! Sounds good - not having to patch would be ideal. Will stay tuned :)