How best to handle ETIMEDOUT?

[rfennell]

This is a follow up to #379, which never got resolved, I have created a new issue as I have now done a lot more research now.

Environment

Node version: v14.15.1 for local dev and I think the same on the Azure DevOps Hosted Agents
Npm version: 6.14.8
OS and version: Azure DevOps Hosted Agents
azure-devops-node-api version: 8.1.1

Issue Description

I have a Node based Azure DevOps Pipeline Extension that generates release notes based on a handlebars template.

• Source
• Marketplace

This task uses the Node SDK to make many calls, commonly well over 100, to the Azure DevOps API to get details of WI, CS, PR and tests associated with a build or release. These results are then injected into a Handlebars template.

In most cases this task works without error, but in some cases my users see a error in the form

Error: connect ETIMEDOUT 13.107.42.18:443

What can I do to handle these timeouts?

Is my only option to abandon the Azure DevOps Node SDK and make all my REST calls natively?

Expected behaviour

The task should reliably complete, irrespective of the number Azure DevOps REST API calls made. The SDK should handle timeouts, retry and throttling of the API.

Actual behaviour

There is intermittent failure, the error being in the form.

Error: connect ETIMEDOUT 13.107.42.18:443

The task could run find multiple times, then fail for a few runs. A retry of a job commonly fixes the immediate problem. It is assumed that the Azure DevOps REST API is being saturated and the SDK retry logic is in adequate.

Steps to reproduce

This is hard to reliable reproduce. It appears to occur more often

• On the hosted agents
• In the late afternoon and evening UK time - when USA based clients are busier?
• When a set of release note contains a lot of associated WI/CS and PRs, hence more API calls to get the details

What I have tried

I have tried all of the following, none have helped

API Retry Settings
I have altered the creation of my WebApi instance to increate the timeouts

 const credentialHandler = getCredentialHandler(pat);
 const options = {
       allowRetries: true,
       maxRetries: 20,
  } as vstsInterfaces.IRequestOptions;
  const organisation = new webApi.WebApi(tpcUri, credentialHandler, options);

Retry on failure
I added a try catch block around all of my SDK API calls. If there was a failure I retired the call (using my own code, the the SDK retry system). Within this retry logic I was able to set the number of retries and the time period to pause before a retry.

This had no effect, it seemed as if once there was an SDK reported timeout no re-connection was possible, even if I paused before the reconnect retry for 60 seconds.

Recreate the WebAPI instance on timeout
I refactored my retry logic to recreate the WebApi object on each retry. This again had no effect, the error still occurred

Logs

The only message seen is

Error: connect ETIMEDOUT 13.107.42.18:443

Examples can be seen in this issue 648 on my Release Notes Task Repo

[jjguijt]

I am encountering the same issue, and agree with the expected behaviour of the library.

[rfennell]

Yes, I have seen a good few reports in various forums.

It would be good to hear if there is a solution to the problem via the API or if it is an underlying constraint of the underlying Azure DevOps REST instances we can do nothing about. Even if that is the case as work around is really needed

[github-actions]

This issue has had no activity in 90 days. Please comment if it is not actually stale

[rfennell]

Still heard nothing to do with this issues, and I still see the problem.

[sommmen]

Still heard nothing to do with this issues, and I still see the problem.

Loving how the bot just closed the issue :sigh:

[sunilsurana]

@rfennell got any clue how to handle this? We also facing the same

[rfennell]

Sorry no, never got a solution

[greengumby]

This just started happening and now I have no release notes. Re-tried the build multiple times and no success. Looks like I have to remove XplatGenerateReleaseNotes?

[jon-freed]

In my case, this error happened consistently on a call to getWorkItem with the expand parameter set to expand relations. However, it wasn't always for the same work item. I'm not sure if getWorkItem with expand relations was the most pertinent factor or if something else was, like like the run time or the number of API calls. Fortunately, setting the WebApi options for retries got me past the error.

[wmcnamara]

Getting this error aswell.

[cjblomqvist]

I recommend reading which might fix your (and others') issue: https://stackoverflow.com/questions/63064393/getting-axios-error-connect-etimedout-when-making-high-volume-of-calls

[wmcnamara]

Just to update; I solved my problem. It was an issue with my proxy config, and was my mistake