Deployment error with no details after updating to Node 8 and firebase-tools@^4.0.0

[jpreynat]

Version info

^4.0.0

Platform Information

Ubuntu 14.04.5 LTS (on Travis CI)

Steps to reproduce

firebase deploy --except hosting with lots of functions (in our case 52) running on Node 8 runtime.

Expected behavior

Before we updated firebase-tools to version 4 and higher (we had the exact same behavior for versions 4.0.0, 4.0.1 and 4.0.2) and using Node 8, updating correctly all of our cloud functions never failed.
We expect it to continue since this happens when deploying our application in production.

Actual behavior

Since this update, there is always at least one or two functions that fail deploying with the following error:

⚠  functions[<our-function-name>(us-central1)]: Deployment error.
Build failed: Build error details not available

During the last deployment running with firebase-tools@4.0.2, 4 functions failed to update.

[laurenzlong]

Hi, I think the issue is likely due to the number of functions you are deploying, and not due to the firebase-tools version (you can of course test this by downgrading). Please deploy a subset of your functions at once instead of all of them, see https://firebase.google.com/docs/cli/#deploy_specific_functions

[mlake]

getting the same error and only deploying 12 functions. Never had these deployment issues until upgrading to node 8.

..if there's a limit to the number you can deploy, shouldn't it be documented somewhere?

[itaydressler]

Same here. Getting it when deploying only one function with node 8.

[laurenzlong]

Ok interesting, thanks for the additional datapoints. So it seems like it's not due to the new versions of firebase-tools per se, but due to deploying to Node 8 vs Node 6. (So using the same firebase-tools but deploying to Node 6 will reduce errors). I've filed a bug internally with the team responsible for the runtime.

[0x80]

This is a pretty annoying bug. My function deployments fail 9 out of 10 times now it seems 😞. Glad to hear it's on peoples radar though. I'm moving back to Node.js 6 for now.

[ahaverty]

@jpreynat @0x80 Does the entire deployment fail or some functions deploy and some fail?
(Investigating whether we should upgrade our 100+ functions to Node v8)

[jpreynat]

@ahaverty In my case, this is only 3 or 4 functions on 52 that fail.
But I'd rather have the deployment fail completely so all our application functions are always on the same version.

[ahaverty]

@jpreynat I agree, we're running deploys via CI, and could live with complete fails but not partial. Thanks for confirming, I'm going to hold off upgrading until this is resolved.

[jpreynat]

This is still happening and it affects our staging and production environments.
We just redeployed the whole application a minute ago, and one of the function failed deploying for no apparent reason.

As @ahaverty also states, the CLI should at least rollback all functions that were successfully deployed to prevent instability in the whole application.
@laurenzlong Do you have any update about the reason of the failure and an ETA for a fix release?
We are about to deploy a critical update to production and need to be sure that it is either fully or not deployed at all, so we can relaunch it and the app won't crash.

[mbleigh]

I would recommend deploying your functions in small groups or individually to minimize potential bad partial deploy states. Rollback is not a feature on the near-term horizon.

…

On Thu, Aug 16, 2018, 10:07 AM Johan Preynat ***@***.***> wrote: This is still happening and it affects our staging and production environments. We just redeployed the whole application a minute ago, and one of the function failed deploying for no apparent reason. As @ahaverty <https://github.com/ahaverty> also states, the CLI should at least rollback all functions that were successfully deployed to prevent instability in the whole application. @laurenzlong <https://github.com/laurenzlong> Do you have any update about the reason of the failure and an ETA for a fix release? We are about to deploy a critical update to production and need to be sure that it is either fully or not deployed at all, so we can relaunch it and the app won't crash. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#853 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAD_gn6e0hkZXlbiM_hv-oBf2T8_Iujks5uRXySgaJpZM4Vsn0E> .

[lukepighetti]

I can confirm that I am also having this issue with Node 8. I will switch back to Node 6.

After switching to Node 8...

1. ~3% invocation failure
2. ~4x increased time to deploy
3. ~80% deploy failure due to 2 of 24 functions failing to deploy

https://stackoverflow.com/questions/52056953/functions-issues-with-node-8-lts/52058796#52058796

I switched to Node 8 to gain access to the spread operator so I could build a library like this. I'm not sure if this is contributing to my lack of success with Node 8.

https://stackoverflow.com/questions/43486278/how-do-i-structure-cloud-functions-for-firebase-to-deploy-multiple-functions-fro/52056785#52056785

[NEO97online]

I'm having the same issue on firebase-tools v4.2.1 using Node 8.

Downgrading back to Node 6 allowed me to deploy.

[mulhoon]

@michaelauderer Same happening here. Seems to be flakey over the past few hours so it might be a cloud issue.

[urkopineda]

@michaelauderer @mulhoon Same here with Node 8.

[yuliankarapetkov]

I have the same issue, but it's weird because these very same functions (4 out of 4) used to deploy successfully yesterday.

[mulhoon]

@yuliankarapetkov Yes, mine were working fine yesterday.
FYI I've been installing node modules with yarn on node v8.9.4, then switching to node v9.4.0 and deploying. Seemed to be going well.

[gautier-gdx]

Same here, but I had this working this morning, I think for our case, this is related to the GCF status right now: https://status.firebase.google.com/incident/Functions/18034 @mulhoon @yuliankarapetkov @urkopineda @michaelauderer

[jojonarte]

woa I thought I'm the only one having this issue. How did you guys solve this?

[jpreynat]

@jojonarte There is no quick fix currently, the current massive errors are due to the Firebase platform encountering a service disruption as stated by @gautier-gdx.
However, we still have problems deploying all our functions at once with Node 8 when Firebase is up.

[lukepighetti]

@jojonarte the only resolution I found was to refactor back to Node 6, deploy an empty index.js to delete all functions (otherwise they remain Node 8 functions), and then deploy the Node 6 index.js

[yuliankarapetkov]

Yes, the problem is that if you have a lot of functions that make use of async/await (or any other Node 8 features), you'll have to spend tremendous amount of time refactoring them.

[lumyus]

Same here. Node 8 tried to deploy 13 functions at once. This issue did not occur yesterday with the same amount of functions.

[nandenjin]

Same problem here. It occurs when tried to update two exist Node 6 functions to Node 8 in this 12 hours.
Using firebase-tools@4.2.1, Solved by deleting exist functions by deploying empty index.js and re-deploying with configuration "engines": { "node": "6" } inside package.json as it was described previous comments.

[deremer]

Any updates on this? This issue is happening for us still as of 9/25. We have 47 functions and anywhere from 0-5 on average will fail to deploy.

[mulhoon]

It's working for me, but this really isn't great for firebase functions.

Does anyone have any good techniques for using a dev environment? e.g. a blue-green deployment to reduce downtime. At the moment I deploy to the live functions, so a deployment failure leads to real app downtime. Thanks

[lukepighetti]

You could blue-green with two Firebase apps.

[kevendra]

+1 Same problem

[radixdev]

Still ongoing.

Build failed: Build error details not available

[selvinfehric]

I am still getting the same error about HIT and MISS on cache stats. This is happening whole day.

Guys, please report deploy issues here, so they might resolve it finally. @pragati27gupta, @taronaeo, @acoyfellow, @MikeWeiZhou
https://firebase.google.com/support/troubleshooter/functions/deploy

[RWOverdijk]

So, firebase uses docker. That's fun to know.

Thanks for reaching out! Your request has been submitted and we'll reach out to you soon.

@selvinfehric has the right idea.

[jaceq]

btw. I opened issue at cloudfunctions issue tracker about that: https://issuetracker.google.com/issues/154260223
Please, comment / star the issue to get more traction there.

[Cdik]

Same here!

[emadalam]

Same here, we have the latest version of firebase tools running on a GitLab runner and this keeps happening. It's such a big annoyance to keep retrying. Our whole CI/CD pipeline is broken due to this, causing such a waste of developer time, tracking the jobs and clicking retry to eternity.

Worst of it, the Firebase Support team hasn't been helpful at all and have pointed to some bunch of random docs, totally ignoring the randomness of function deployment errors.

[oviliz]

Happens to us too, when locally on Ubuntu 16.04 with firebase-tools 7.12.1 or via Azure DevOps Task with the latest 8.1.1.
Ridiculous that this issue is closed and nobody (Google side) seems to do care less.

[neilpoulin]

this is happening to us, too. We've been battling it for nearly a year. We only have about 30 functions (of all types, https, firestore triggers, pub sub subscribers, auth triggers). It mostly happens when we deploy via CI/CD (circleci), but it happens sometimes when we deploy from our local machines, too (although we do it so infrequently it's hard to say).

I'm running node 8, because of a bug firebase had a few months ago with node 10. Typically, probably 8 or 9 times out of 10, one or two of our functions will just fail to deploy. It seems to be a random one each time - never a consistent failure. This happens without making any changes to our functions code - we may just be deploying a front-end change (via hosting, but we want to keep versions in sync across files via hosting and the API). So, we know it's not our code.

I finally bit the bullet today and spend half of the day breaking out our "large number" (~30) of functions into smaller groups that deploy on their own, with each group having less than 10. THIS SITLL DOES NOT WORK. I've had several failures in a row. in my first group of functions, which only has 6 items. This "group" is actually declared using the firebase deploy --only functions:func1,functions:func2,...functions:func6 syntax, not a group defined in the main export.

[chenlevy]

Same here
Deployment error.
Anywhere between 1-5 different functions fail to deploy:
Build failed: {"cacheStats": [{"status": "MISS", ...

[mbleigh]

Anyone experiencing this error please +1 and weigh in on this from the public Cloud issue tracker: https://issuetracker.google.com/issues/154260223

…

On Sat, Apr 25, 2020, 4:49 AM chenlevy ***@***.***> wrote: Same here Deployment error. Anywhere between 1-5 different functions fail to deploy: Build failed: {"cacheStats": [{"status": "MISS", ... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#853 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAH7RZHEUCY2EAR2ABU3DROLE23ANCNFSM4FNSPUCA> .

[erykos111]

I have found that, when creating new project, for my previous one I didn't have the "google analytics" enabled, and every single of my deployments was failing. I just tested it with brand new project and everything works, when I do enable google-analytics

[mbleigh]

Whether Analytics is enabled or not will not affect this error, that's likely just been a coincidence.

…

On Sat, Apr 25, 2020, 12:17 PM erykos111 ***@***.***> wrote: [image: image] <https://user-images.githubusercontent.com/47335304/80288700-94ee1680-8731-11ea-8f57-66598893458e.png> I have found that, when creating new project, for my previous one I didn't have the "google analytics" enabled, and every single of my deployments was failing. I just tested it with brand new project and everything works, when I do enable google-analytics — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#853 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAH7TAOV2IOQG7NUDBXFTROMZMHANCNFSM4FNSPUCA> .

[taronaeo]

As stated in the above comment, enabling Google Analytics does not change the failed deployment of functions. What you have probably done is created a new project in a server that does not have their Cloud Functions server failing to operate.

[dudemeister]

Just ran into the same issue - deployments were working fine until we got these, reliably on two functions - started sometime today:

⚠ functions[legacyFeedRedirect(us-central1)]: Deployment error.
Build failed: {"cacheStats": [{"status": "MISS", "hash": "526b4459b8b6ae9cc7de4bbed53b2d24a8c5fcf1737f07f27a44e4029d472306", "type": "docker_layer_cache", "level": "global"}, {"status": "HIT", "hash": "526b4459b8b6ae9cc7de4bbed53b2d24a8c5fcf1737f07f27a44e4029d472306", "type": "docker_layer_cache", "level": "project"}]}
⚠ functions[updateUserOnboardingFlags(us-central1)]: Deployment error.
Build failed: {"cacheStats": [{"status": "MISS", "hash": "526b4459b8b6ae9cc7de4bbed53b2d24a8c5fcf1737f07f27a44e4029d472306", "type": "docker_layer_cache", "level": "global"}, {"status": "HIT", "hash": "526b4459b8b6ae9cc7de4bbed53b2d24a8c5fcf1737f07f27a44e4029d472306", "type": "docker_layer_cache", "level": "project"}]}

[jpreynat]

To anyone that still encounter deployment errors:

After first fixing our deployments by switching on Node 10 and running with the latest version of firebase-tools, we recently (about 2 weeks ago) encountered errors again with an opaque Build error: details not available. We were completely stuck because this happened constantly for all of our functions on all our environments.

After investigating, we noticed that the build time for cloud functions had been reduced on GCP from 10 minutes to only 5 minutes.

To improve build time, GCP uses caching to re-install node_modules based on the hash of either the package-lock.json or the yarn.lock files, depending on which you are using.
But if your dependencies are updated and this file changes, GCP has to re-install all of them, which was taking too long in our case, especially since we have many native modules that need to be rebuilt. I think this is what we can see in the Node 8 error details with the "status": "MISS".

We ended up building our own tooling to compile our functions separately, bundling all non-native modules in our main function code using webpack, and having a trimmed package.json file for each function with only the missing dependencies for each of them.

Now our deployments work well every time, and take 1 to 2 minutes max per function.

[nes123]

+1

[TranDieuLinh]

+1
Same error.
Build failed: {"cacheStats": [{"status": "MISS", "hash": "8c20b0a9a3a6bd52591da4f39acf1b59e56da60640e148a5d96a494617066c6d", "type": "docker_layer_cache", "level": "global"}, {"status": "HIT", "hash": "8c20b0a9a3a6bd52591da4f39acf1b59e56da60640e148a5d96a494617066c6d", "type": "docker_layer_cache", "level": "project"}]}

[fatihacet]

We have been seriously impacted by this issue and were unable to deploy since last week. Details can be found in my comment on Google Issue Tracker.

We come up with a custom deployment script which retries deployment of failed Cloud Functions. Leaving it here, so it may give you an idea, or you can use it directly. We already deployed to Staging environment more than 20 times using this script.

Hope this helps someone until this issue is fixed.

[jpreynat]

@fatihacet As I mentioned earlier, the build errors mostly happen because of a timeout during the deploy process, which has been reduced from 10 to 5 minutes by GCP about 2 weeks ago.

We also had a retry system in place, implemented in a fork of firebase-tools (and usable) here (PR for official firebase-tools is here #1977).

However, we had been stuck with our deployments for more than 2 weeks due to the recent timeout decrease, and retries wouldn't solve the problem for us, as we couldn't deploy a single function on any of our environments, even with 3 to 5 retries, during this period.

The only fix is to reduce the amount of dependencies that you are using for your cloud functions.
In our case, we had to compile them separately and bundle all that could be in the function's code using webpack.

With this update on the build system timeout, the firebase model of using a single entry point for all of the functions to be deployed simply doesn't work anymore.

[Trinhlvtq]

I just updated firebase-cli version 8.2.0
However, I continue to encounter this error
Build failed: {"cacheStats": [{"status": "MISS", "hash": "8c20b0a9a3a6bd52591da4f39acf1b59e56da60640e148a5d96a494617066c6d", "type": "docker_layer_cache", "level": "global"}, {"status": "HIT", "hash": "8c20b0a9a3a6bd52591da4f39acf1b59e56da60640e148a5d96a494617066c6d", "type": "docker_layer_cache", "level": "project"}]}

[jpreynat]

@Trinhlvtq, upgrading firebase-tools won’t solve this issue.

Out of curiosity, how many node dependencies do you use in your project?
Do you use native ones in your project?

[sunnixx]

We've been having this issue for the last 2 weeks now.
In our case we only get
build failed: build error details not available
and when we check the logs on GCP we see a status code of 13.

[taronaeo]

@sunnixx This issue has already been resolved in Google's Issue tracker #154260223. If you think that your problem is related to this issue, please post your issue there.

[devth]

This just started for me today. Node 10. No clue what caused it but every attempt to deploy functions from CI results in this error.

[taronaeo]

@devth Please copy & paste the output here, so that the team can understand what is going on.

[devth]

@taronaeo here's the output:

i  functions: updating Node.js 10 function onCreate(us-central1)...
 i  functions: updating Node.js 10 function createUser(us-central1)...
 i  functions: updating Node.js 10 function onDocCreate(us-central1)...
 i  functions: updating Node.js 10 function onSub1DocCreate(us-central1)...
 i  functions: updating Node.js 10 function onSub2DocCreate(us-central1)...
 i  functions: updating Node.js 10 function next(us-central1)...
 ⚠  functions[createUser(us-central1)]: Deployment error.
 Build failed: Build error details not available.
 ⚠  functions[onCreate(us-central1)]: Deployment error.
 Build failed: Build error details not available.
 ⚠  functions[onSub1DocCreate(us-central1)]: Deployment error.
 Build failed: Build error details not available.
 ⚠  functions[next(us-central1)]: Deployment error.
 Build failed: Build error details not available.
 ⚠  functions[onSub2DocCreate(us-central1)]: Deployment error.
 Build failed: Build error details not available.
 ⚠  functions[onDocCreate(us-central1)]: Deployment error.
 Build failed: Build error details not available.
 Functions deploy had errors with the following functions:
 	createUser
 	next
 	onCreate
 	onDocCreate
 	onSub1DocCreate
 	onSub2DocCreate
 To try redeploying those functions, run:
     firebase deploy --only functions:createUser,functions:next,functions:onCreate,functions:onDocCreate,functions:onSub1DocCreate,functions:onSub2DocCreate
 To continue deploying other features (such as database), run:
     firebase deploy --except functions
 Error: Functions did not deploy properly.
 error Command failed with exit code 1.

Update 1: It seems to fail consistently with firebase deploy. If I instead firebase deploy --only functions it will sometimes work / sometimes fail with the same error.
Update 2: well today it seems to just fail all the time.

Thanks.

[jpreynat]

@devth have a look at this message.

[devth]

So apparently it has to do with too many deps. I'm running Nextjs on Firebase Functions, so I need my entire web stack's deps. This is a fairly small and new app too, so it's only going to grow.

Side note: this problem has broken my current functions, which took down my site (everything just 404s now). Seems like a major problem. If this was prod I'd be fully down with no solution.

[mbleigh]

Hi folks, if you are experiencing this error please search for a similar bug or file a new bug on the public issue tracker for Cloud Functions. This issue has already been closed and we won't be able to help you resolve it.