Intermittent errors in deploy error handling

[ilkka]

We are deploying a Netlify site as part of a Jenkins build, and sometimes are seeing what look like network connection issues between our CI provider and Netlify. The troublesome aspect here is that Jenkins fails to flag these builds as failed, thinking that they've succeeded regardless of this error output at the bottom of the console log:

+ netlify deploy -t deadbeef .
Deploying folder: .
{ client: 
   { access_token: 'acdc1234',
     client_id: 'abba4321',
     client_secret: undefined,
     redirect_uri: undefined,
     ENDPOINT: 'https://api.netlify.com',
     VERSION: 'v1',
     hostname: 'api.netlify.com',
     ssl: [ 'https://', index: 0, input: 'https://api.netlify.com' ],
     port: 443,
     http: 
      { Server: [Object],
        createServer: [Function],
        globalAgent: [Object],
        Agent: [Object],
        request: [Function],
        get: [Function] } },
  data: 
   { Error: socket hang up
       at createHangUpError (_http_client.js:250:15)
       at TLSSocket.socketOnEnd (_http_client.js:342:23)
       at emitNone (events.js:91:20)
       at TLSSocket.emit (events.js:185:7)
       at endReadableNT (_stream_readable.js:926:12)
       at _combinedTickCallback (internal/process/next_tick.js:74:11)
       at process._tickCallback (internal/process/next_tick.js:98:9) code: 'ECONNRESET' },
  meta: null }
Potentially unhandled rejection [3] SyntaxError: Unexpected token E in JSON at position 0
    at Object.parse (native)
    at Object.exports.log (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/lib/helpers/error_logger.js:11:21)
    at /data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/lib/commands/deploy.js:144:17
    at tryCatchReject (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/node_modules/when/lib/makePromise.js:845:30)
    at runContinuation1 (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/node_modules/when/lib/makePromise.js:804:4)
    at Rejected.when (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/node_modules/when/lib/makePromise.js:625:4)
    at Pending.run (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/node_modules/when/lib/makePromise.js:483:13)
    at Scheduler._drain (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/node_modules/when/lib/Scheduler.js:62:19)
    at Scheduler.drain (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/node_modules/when/lib/Scheduler.js:27:9)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)

The "error handling error" seems to be trying to parse error.data, containing the string "Error: ..." in error_logger.js. Don't know why that would cause an exit with return code 0 though, or if there are more Jenkins shenanigans actually causing that.

[schickling]

Same here when running via CircleCI. Unfortunately this blocks us from using Netlify.

[schickling]

Any idea @biilmann @cassiozen?

[biilmann]

I pushed an update to netlify-cli yesterday that mainly pulls in an updated version of the node client library. The fix in the node client was related to deploys with many concurrent uploads failing with that kind of error. Can you make sure you're on netlify-cli v1.0.3 and let me know if that version still has the problem?

[schickling]

Hey @biilmann. I've installed the netlify-cli today, so it's using version 1.0.3. Could there be a different issue?

[nolessafool]

ah, I filed #13 today - to give us some better debugging tools around this same issue.

[schickling]

Any update on this? 80% of our builds are seeing this error...

[nolessafool]

Hi @schickling and @ilkka - I think we just fixed the longer term problem with larger deploys. Can you give v1.1.0 a spin and let us know how it goes?

[ilkka]

Thanks, we'll bump our dependency and see how it goes!

[nolessafool]

OK, this is still an issue with version 1.1.0 and I've now seen it firsthand and think we (@netlify) will be able to reproduce well enough to debug. Is there anyone attached to this issue (mainly not sure about @ilkka ) who isn't using CircleCI when they get the error?

It felt to me like the "working" deploys completed at least an order of magnitude faster than the failing ones, so while I'm not suggesting this as a long-term workaround or a fix, you may get some mileage out of a stopgap with something like the first answer here to restart a failed deploy rather than having to redo the build:

http://stackoverflow.com/questions/5161193/bash-script-that-kills-a-child-process-after-a-given-timeout

[nolessafool]

We've hit some walls in debugging this and want to take it to CircleCI - but are not paying customers.

Does one of you who is a paying customer mind writing in with a real ticket (rather than a forum post which is all we can do for free) and cc:ing myself (chrism@netlify.com) and our buildmaster (ryan@netlify.com) so we can present our evidence and ask them for help? After that point we probably don't actually need you to spend any time at it, just trying to get in the door :)

If someone does it please comment here so we don't hit them multiple times in parallel. (cc @schickling and @ilkka)

[nolessafool]

hmm, haven't heard back from anyone yet but would like to make some progress on this - @schickling any chance you can email Circle and cc: us?

[schickling]

Sorry we're also not on the needed pricing plan and they haven't responded to our previous inquiries yet.

[ilkka]

Yeah, sorry, we're still only freeloaders on CircleCI too.