Security handshake failed and speech recognition stops working

[shoten]

Description
I encountered this random bug on the smart mirror project, but it seems to come from sonus.
When I try the sonus test command (see below)

case 1 : I say the hotword sonus detects it, I wait a while and sometimes if I say nothing I have an handshake error.

case 2 : I say the hotword sonus detects it, I wait a while and I say "hello" or something and then I have an handshake error.

case 3 : I say the hotword sonus detects it, I speak, everthing is going well, I wait a while and I say the hotword then I have an handshake error.

After that it can detect the hotword but it doesn't detect the speech anymore.
Everytime I have at least 3 handshake errors

Unfortunalty it s not easy to reproduce, even if it occurs very often.

How to reproduce

pi@raspberrypi:~/smart-mirror $ npm run sonus

> smart-mirror@0.0.7 sonus /home/pi/smart-mirror
> node sonus.js

!h: 2
E1227 00:11:15.508035697    1888 handshake.c:128]            Security handshake failed: {"created":"@1482793875.507858875","description":"Handshake read failed","file":"../src/core/lib/security/transport/handshake.c","file_line":237,"referenced_errors":[{"created":"@1482793875.507826480","description":"FD shutdown","file":"../src/core/lib/iomgr/ev_epoll_linux.c","file_line":948}]}
E1227 00:11:17.511256783    1885 handshake.c:128]            Security handshake failed: {"created":"@1482793877.511095482","description":"Handshake read failed","file":"../src/core/lib/security/transport/handshake.c","file_line":237,"referenced_errors":[{"created":"@1482793877.511069961","description":"FD shutdown","file":"../src/core/lib/iomgr/ev_epoll_linux.c","file_line":948}]}
E1227 00:11:18.513975132    1885 handshake.c:128]            Security handshake failed: {"created":"@1482793878.513764665","description":"Handshake read failed","file":"../src/core/lib/security/transport/handshake.c","file_line":237,"referenced_errors":[{"created":"@1482793878.513719300","description":"FD shutdown","file":"../src/core/lib/iomgr/ev_epoll_linux.c","file_line":948}]}
E1227 00:11:19.516665305    1885 handshake.c:128]            Security handshake failed: {"created":"@1482793879.516573118","description":"Handshake read failed","file":"../src/core/lib/security/transport/handshake.c","file_line":237,"referenced_errors":[{"created":"@1482793879.516554004","description":"FD shutdown","file":"../src/core/lib/iomgr/ev_epoll_linux.c","file_line":948}]}

Configuration

pi@raspberrypi:~/smart-mirror $ ulimit -n
65536
pi@raspberrypi:~/smart-mirror $  node -v
v6.9.2
pi@raspberrypi:~/smart-mirror $ npm -v
3.10.9

Comments
I have this error on the smart-mirror master branch and on the dev branch too
I make sure to clean up everything with "git clean -xdf -e config.js" and "npm install" each time

Ideas
After some googling, some grpc users seems to have the same issues and it looks related to a connection limit but I only have the smart-mirror running ... so it's weird
grpc/grpc#7985

Thanks in advance for your help !

[evancohen]

Thanks for the detailed issue! I've seen similar problems when the Pi's clock is set incorrectly: evancohen/smart-mirror#468

According to the @google-cloud/speech documentation there isn't anything explicit that you have to do to "close" the connection - but if the issue is truly connection limits then that's a potential cause.

I haven't personally experienced this issue, so it could be due to an update of the grpc library (we've had issues with dependencies breaking us in the past). It seems that there are other platforms having similar issues: grpc/grpc#8732

To complicate things further, it seems that @google-cloud/speech and its dependencies actually require 2 different versions of grpc (at least in my install):

> npm ls sonus grpc
sonus@0.1.2 /Users/evan/Git/sonus
└─┬ @google-cloud/speech@0.2.0
  ├─┬ @google-cloud/common@0.6.0
  │ └── grpc@1.0.0
  └─┬ google-gax@0.6.0
    └── grpc@0.15.0

I've got a pretty outdated version of @google-cloud/speech. You might try to manually install the version I'm using and see if that fixes the problem.

In the meantime I'm going to install the latest version of the library and see if I can reproduce this.

[shekit]

I got this same error, however it's the first time it has happened. Speech recognition stopped after that. This also happened in an app that was already running and not on startup. It's originating in the child process. Restarting the app made it work again.

Using sonus v0.1.5 in an electron app(v1.4.14)

LISTEN PROCESS ERROR 0 16:00:42.108220711 16560 handshake.c:128] Security handshake failed: {"created":"@1484946042.108119982","description":"Handshake read failed","file":"../src/core/lib/security/transport/handshake.c","file_line":237,"referenced_errors":[{"created":"@1484946042.108087014","description":"FD shutdown","file":"../src/core/lib/iomgr/ev_epoll_linux.c","file_line":948}]}

Any luck in recreating this error?

[shekit]

I've been able to consistently recreate this error. I've tried this on four wifi networks A, B, C & D:

1. On Network A, every time I start the app and say the keyword followed by the command the very first time, I get anywhere between 1-3 handshake errors in quick succession. Speech detection continues to work in about 80% of the cases (just a guestimate) or requires the app to be restarted.

2. On Network B, I don't get the handshake errors on startup. The internet is working, but nothing seems to work after the hotword getting detected, no google speech part.

3. On Network C & D, no handshake errors and speech detection seems to be working, though I still need to keep this running for a longer time to confirm.

Looks like it might be a network issue of some sort?

[evancohen]

This is an issue with the google speech library, more specifically with the way that grpc handles ongoing connections: googleapis/google-cloud-node#1905

Seems like the workaround is to extend the time allowed when trying to reestablish a connection. This is something that doesn't require the modification of sonus to fix!

I'm taking a bit of a guess here because I only glanced at the code, but if you construct your speech object like this before you pass it to Sonus it might fix your issue:

const speech = require('@google-cloud/speech')({
  projectId: 'streaming-speech-sample',
  keyFilename: ROOT_DIR + 'keyfile.json',
  "grpc.initial_reconnect_backoff_ms": 10000
})

[evancohen]

I believe this issue is now resolved. If it isn't I can reopen :)