Meteor 1.8.0.2: TypeError: Cannot set property 'setWebsocketTimeout' of null

[c0desolut1ons]

Version of Meteor: 1.8.0.2.
Operating System: Windows Server 2012, Linux Centos

When we deploy and start meteor app we log error:

Meteor log:

TypeError: Cannot set property 'setWebsocketTimeout' of null
    at Server.<anonymous> (packages/ddp-server/stream_server.js:99:5)
    at emitOne (events.js:116:13)
    at Server.emit (events.js:211:7)
    at App.emit (c:\cmms.bee\prod\meteor\bundle\programs\server\npm\node_modules\meteor\ddp-server\node_modules\sockjs\lib\sockjs.js:196:29)
    at c:\cmms.bee\prod\meteor\bundle\programs\server\npm\node_modules\meteor\ddp-server\node_modules\sockjs\lib\transport.js:111:25
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)

Direct file link:

meteor/packages/ddp-server/stream_server.js

Line 99 in f04160a

socket.setWebsocketTimeout = function (timeout) {

More info:
SockJS (maybe) related issue: sockjs/sockjs-node#121
The right way for a client to reconnect after the server does a socket.disconnect( true )
socketio/socket.io#2476

Is there any way how to make this work?
Thanks.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kombadzomba]

Please solve this. Don't just ignore it and close it like it never happened. No one from Meteor team has ever responded to this issue at all. We are still facing this problem in production and have serious meteor issues since this started. There was no response on this for a long time so we started to make plans on substituting meteor with something else. Please don't make us speed that process up.

[filipenevola]

Hi @kombadzomba thanks for bringing this up. I'm Meteor Evangelist working for Meteor Software (the new Tiny company behind Meteor).

I'm also a Meteor user for a long time and I never saw this error in my apps, maybe you have something different in your application.

Can you create somehow a reproduction? I would like to understand this issue in depth.

We are committed to Meteor (see the roadmap published today) and you are not going to need to substitute Meteor, at least not because of issues 😉

I already marked this issue as confirmed then stale bot is going to ignore it.

[kombadzomba]

Hi @filipenevola and thanks for kind human response. We didn't find a way to reproduce it. However, when it hapends node process crashes. We had to pm2 the node processes in order to keep them alive. Here is the link to details on the related subject so you could get deeper into understanding it: Urigo/meteor-client-bundler#96 (comment)

[filipenevola]

hi @kombadzomba the only reason that can be happening is on('connection') receiving an undefined connection. The only similar issue I found was this one sockjs/sockjs-node#90 but this fix is exactly in the version of sockjs used by Meteor (0.3.19) then I think in your case it's a different root cause.

Have you tried to run Meteor from a fork adding this line below:

if (!socket) {
  console.warn('connection not available on server on connection');
  return;
}

before this line (the line where the error is happening):

meteor/packages/ddp-server/stream_server.js

Line 99 in b9f2bb2

socket.setWebsocketTimeout = function (timeout) {

I believe this is not going to cause any issue and probably on will be called again with the proper connection.

If you didn't try this yet I would suggest to you:
1 - Try this using in a fork of Meteor.
2 - If this fixes your problem, create a PR with this addition.

Please let me know if this plan sounds good to you.

[kombadzomba]

Thanks for your effort. However, you are suggesting a quick patch while believing it won't cause any further issues asking me to test your believes. However, as I mentioned, we have already patched this thanks to pm2. With all my respect, I would rather preffer a thorough approach in solving this issue without trial and error approach.

[brianlukoff]

@filipenevola We are now seeing this issue happen seemingly at random in production on Meteor 1.9 (an app upgraded from 1.8.2 that never experienced this before). It is happening not on startup but at some point while the server is running.

[dovydaskukalis]

Same when upgrading from 1.8.2 to 1.9.

TypeError: Cannot set property 'setWebsocketTimeout' of null
    at Server.<anonymous> (packages/ddp-server/stream_server.js:99:5)
    at Server.emit (events.js:223:5)
    at Server.EventEmitter.emit (domain.js:475:20)
    at App.emit (/var/www/app/programs/server/npm/node_modules/meteor/ddp-server/node_modules/sockjs/lib/sockjs.js:196:29)
    at /var/www/app/programs/server/npm/node_modules/meteor/ddp-server/node_modules/sockjs/lib/transport.js:111:25
    at processTicksAndRejections (internal/process/task_queues.js:76:11)

[Discordius]

We are also seeing this issue happening after upgrading to 1.9. Same stacktrace as above:

TypeError: Cannot set property 'setWebsocketTimeout' of null
    at Server.<anonymous> (packages/ddp-server/stream_server.js:99:5)
    at Server.emit (events.js:210:5)
    at Server.EventEmitter.emit (domain.js:475:20)
    at App.emit (/var/app/current/programs/server/npm/node_modules/meteor/ddp-server/node_modules/sockjs/lib/sockjs.js:196:29)
    at /var/app/current/programs/server/npm/node_modules/meteor/ddp-server/node_modules/sockjs/lib/transport.js:111:25
    at processTicksAndRejections (internal/process/task_queues.js:75:11)

[jimrandomh]

After some more investigating with @Discordius, this looks like mostly a bug in sockjs-node, though it could be mitigated by some defensive programming in meteor:ddp-server. Specifically, https://github.com/sockjs/sockjs-node/blob/master/lib/session.js#L57 puts a call to server.emit('connection', this.connection); in a call to nextTick, but this.connection may be set to null from didTimeout.

We don't know exactly what circumstances will trigger this, but likely cases are if either something else in the node event loop takes long enough to make this time out, or if a nonstandard client opens a connection and closes it immediately without completing all of the handshake. If it's the latter, this would explain why it's hard to replicate in a debugging environment: the most likely source of that nonstandard client behavior would be from crawlers.

Regardless of what actually triggers it, putting a check for null in either sockjs-node or meteor:ddp-server should fix it, and having looked at this area of the code a bunch, I don't expect any side effects from fixing it that way.

We're getting this error on our production servers several times per hour, so after we deploy with a forked version of sockjs, we should be able to report whether it worked.

[markreid]

This has also started happening to us since updating to 1.9.0. Can't determine exactly what's causing it at this point but it looks to me like @jimrandomh's defensive code in meteor:ddp-server would do the trick. As to why it's happening, appears to be a sockjs issue that's been around for several years without anybody really getting to the bottom of it...

[filipenevola]

I just published ddp-server@2.3.1 fixing this issue.