Errors deploying functions: Server Error. Client network socket disconnected before secure TLS connection was established

[xaphod]

[REQUIRED] Environment info

firebase-tools: 9.1.0

Platform: macOS 10.15.7 (catalina)

[REQUIRED] Test case

Attempt to deploy a firebase project that has hosting, functions, firestore

[REQUIRED] Steps to reproduce

firebase deploy

[REQUIRED] Expected behavior

Deploys correctly

[REQUIRED] Actual behavior

Fails with different error messages, mostly like
⚠ functions[onCreateUser3(us-central1)]: Deployment error. Server Error. Client network socket disconnected before secure TLS connection was established

Also saw:
Error: Failed to get status of functions deployments

I spent over an hour resetting my internet connection (modem), connecting via ethernet instead of wifi, etc.
Eventually I tried downgrading from firebase-tools 9.1.0 to 8.16.2 and it instantly fixed the problem -- after over 10 failures with 9.1.0

[xaphod]

Update: now i got an error with 8.16.2 too:
Error: Could not read source directory. Remove links and shortcuts and try again.

I just tried again and it worked the second time.
This unreliability is really unnerving when you're migrating between function versions and you have no idea what is actually in production when an error kills everything...

[xaphod]

Update 2: now I got Server Error. Client network socket disconnected before secure TLS connection was established again with 8.16.2, so this isn't a new issue I suppose.

But I still really need to get this working....

[xaphod]

Some debug logs...

[debug] [2021-01-02T23:00:24.343Z] >>> HTTP REQUEST GET https://cloudfunctions.googleapis.com/v1/projects/<redacted>/locations/-/functions

[debug] [2021-01-02T23:00:34.538Z] [functions] failed to list functions for <redacted>
[debug] [2021-01-02T23:00:34.538Z] [functions] Server Error. Client network socket disconnected before secure TLS connection was established
[debug] [2021-01-02T23:00:34.649Z]
[error]
[error] Error: An unexpected error has occurred.

[xaphod]

I am wondering if this is a problem with my internet service provider: when I use hotspot on my cell phone, this error doesn't seem to appear (but n=small, not sure if this is just luck).

Anyone else experiencing this issue?

[samtstern]

@xaphod I do think this could be an issue with your ISP. I have run into the same issue on a few of my personal projects and I have only been able to solve it by attempting the deploy by SSHing into a Cloud VM (so a different internet connection).

However I think what we need to do is change the error message:

Error: Could not read source directory. Remove links and shortcuts and try again.

That's clearly not accurate.

[samtstern]

Let's consolidate this issue under a slightly earlier duplicate:
#2961

[xaphod]

Hey @samtstern, I keep getting this error all over the place. The other issue you've duped this to is different: it's about the masking, not the fact that there are spurious network errors.

I've been getting these network errors only with firebase for the last month, consistently... whereas nothing else has ever had a single failure. I've tried everything I can think of to rule out errors from my home, my ISP etc -- AFAICS this is a software issue or this is a server issue...

Can we reopen this?

[debug] [2021-01-20T02:00:06.330Z] >>> HTTP REQUEST GET https://firebase.googleapis.com/v1beta1/projects/sharedgallery-staging/adminSdkConfig

[debug] [2021-01-20T02:00:20.528Z] Error: Client network socket disconnected before secure TLS connection was established
    at connResetException (internal/errors.js:609:14)
    at TLSSocket.onConnectEnd (_tls_wrap.js:1549:19)
    at Object.onceWrapper (events.js:420:28)
    at TLSSocket.emit (events.js:326:22)
    at TLSSocket.EventEmitter.emit (domain.js:483:12)
    at endReadableNT (_stream_readable.js:1241:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)
[error]
[error] Error: Server Error. Client network socket disconnected before secure TLS connection was established

[samtstern]

@xaphod I'm inclined to agree with you because I've had more network issues with firebase-tools than other things on my personal computer although I have absolutely no idea where to begin this investigation. As you could imagine this could be anything from a Node library issue to something deep in Google's network stack and most of us here on the CLI repo are wholly unqualified to investigate anything in the latter category.

[samtstern]

@bkendall can you think of any useful investigation we could do here? If not is there somewhere internally we could report this at least?

[bkendall]

What version of Node are you using?

Could you run the command with NODE_DEBUG=socket,net,http before it? E.g. NODE_DEBUG=socket,net,http firebase projects:list. Something seems to be up with the node stack, so maybe that could have some useful output. If you don't want to clean the output up a ton (there's tokens and names all up in there I'm sure), you can send it to me directly at bkend [@] google[.]com and I'll take a look (reference this bug and your GH name so it catches my attention).

[xaphod]

Hi @bkendall I emailed you some logs with the NODE_DEBUG you requested. It's node v12.20.0
thanks

[xaphod]

@samtstern I'm trying to get this issue further investigated on multiple sides (Google via bkendall, my ISP, others). Your earlier comment on Jan 20 indicates that you have experienced the same issue. If that is true, would you mind sharing the rough location where you encountered these errors (ie. city if you're comfortable, state/province if not), and what source of internet (ie. provider) you were using at the time?

[samtstern]

@xaphod sure I am in London, UK using a 5g home internet hub and Vodafone is my service provider.

[xaphod]

Thanks Sam - and are you seeing the same TLS/SSL errors (ie. Client network socket disconnected) or something different?

…

On Thu, Feb 11, 2021 at 11:33 AM Sam Stern ***@***.***> wrote: @xaphod <https://github.com/xaphod> sure I am in London, UK using a 5g home internet hub and Vodafone is my service provider. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2988 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4Z6LSTGCTE4PFQAAI23G3S6QBEVANCNFSM4VRJDQNA> .

[samtstern]

@xaphod I see these kinds of errors all the time but it's not Firebase specific. I've seen them with the GitHub API and with basically any other network request that I've bothered to monitor.

[bkendall]

This is now quite dated, and I don't think we ever made any good progress figuring out what was happening. Network issue debugging is hard, unfortunately.

Hopefully this has cleared up for you in the meantime!

[TheRealMikeD]

Hey all. I am getting a similar issue from within an app running in production on Cloud Functions.
In production, I am using the bountyrush library for interacting with Firestore rather than the Firebase library. When running simple queries, I sometimes get the following error:
request to https://firestore.googleapis.com/v1/projects/[projectname]/databases/(default)/documents:runQuery failed, reason: Client network socket disconnected before secure TLS connection was established
The fact that this error occurs from within the Cloud Functions environment would seem to indicate (to me, anyway) that it's not necessarily anything peculiar to your development environment.

If I keep seeing these errors, I may have to stop using the bountyrush library, even though I suspect that the fault lies with the Firestore API, not the bountyrush lib itself.