Random OperationCanceledException on Uri checks

[mdalepiane]

This may not be a bug, but I need help troubleshooting this issue.

What happened:
Our health checks are called every minute or so and we are randomly receiving an OperationCanceledException on some UrlGroup checks and when this happens the dependency reports as unhealthy. It normally happens after around 3 seconds. I need help to understand what is going on. We suspect network issues may be the root cause but this exception gives no insights on what's going on.

OperationCanceledException: Connection id ""0HLV847B7BG38"", Request id ""0HLV847B7BG38:00000001"": An unhandled exception was thrown by the application.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService.CheckHealthAsync(Func`2 predicate, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Diagnostics.HealthChecks.HealthCheckMiddleware.InvokeAsync(HttpContext httpContext)
   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at NewRelic.Providers.Wrapper.AspNetCore.WrapPipelineMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

What you expected to happen:
We expect a way to understand what is causing this error. We need some insight about why this exception is thrown.

How to reproduce it (as minimally and precisely as possible):
It seems to be random and related to a specific environment.

Environment:

• Application runs on docker using mcr.microsoft.com/dotnet/core/aspnet:2.2-stretch-slim
• AspNetCore.HealthChecks.AzureServiceBus - 2.2.5
• AspNetCore.HealthChecks.SqlServer - 2.2.1
• AspNetCore.HealthChecks.UI - 2.2.20
• AspNetCore.HealthChecks.Uris - 2.2.3

[mdalepiane]

I've found this unresolved question in StackOverflow that seems to be the same problem:

Asp.net core healthchecks randomly fails with TaskCanceledException or OperationCanceledException

[unaizorrilla]

Hi @mdalepiane

• I can help on debug this, typocally HttpClient throw this on timeout or when the services does not response ( HttpClient throws TaskCanceledException on timeout dotnet/runtime#21965 ) I try to review the code and check if we can throw with more insights about the real problem.

• The provided link is not related with this project, is related with abstractions on ASP.NET Core. Probably this will be filled on https://github.com/dotnet/aspnetcore

I close the issue but create a new item for improving insights on http client exceptions