ASP.NET app becomes unresponsive after days of running. Logs full with exceptions from Prometheus.LabelSequence..ctor

[JeremyMahieu]

Using prometheus-net.AspNetCore 8.0.1 in a .NET 7.0 application (7.0.9). Application runs for many days without problems until it becomes unresponsive. We have FirstChanceException events logged, so this is the reason why we're seeing the exception. The logs are riddled with them, however we cannot be sure this is the reason the site becomes unresponsive.

We use MapMetrics and no otherwise no custom metrics or anything.

app.UseEndpoints(endpoints =>
{
   (...)
    endpoints.MapMetrics(); // Prometheus
});

// Prevent memory issue with prometheus
Metrics.SuppressDefaultMetrics(new SuppressDefaultMetricOptions
{
    SuppressEventCounters = true
});

[2023-08-29 12:11:39Z DBG] FirstChanceException event raised in [redacted]
System.ArgumentException: The list of label values must have the same number of elements as the list of label names.
   at Prometheus.LabelSequence..ctor(StringSequence names, StringSequence values)
[2023-08-29 12:11:39Z DBG]    at [redacted].<>c.<LogFirstChanceExceptions>b__1_0(Object source, FirstChanceExceptionEventArgs e)
   at Prometheus.LabelSequence..ctor(StringSequence names, StringSequence values)
   at Prometheus.LabelSequence.From(StringSequence names, StringSequence values)
   at Prometheus.Collector`1.Labels(String[] labelValues)
   at Prometheus.Collector`1.WithLabels(String[] labelValues)
   at Prometheus.ManagedLifetimeMetricHandle`2.WithLease(Action`1 action, String[] labelValues)
   at Prometheus.MeterAdapter.OnMeasurementRecorded[T](Instrument instrument, T measurement, ReadOnlySpan`1 tags, Object state)
   at System.Diagnostics.Metrics.Instrument`1.RecordMeasurement(T measurement, ReadOnlySpan`1 tags)
   at System.Diagnostics.Metrics.Instrument`1.RecordMeasurement(T measurement, TagList& tagList)
   at System.Diagnostics.Metrics.UpDownCounter`1.Add(T delta, TagList& tagList)
   at System.Net.Http.Metrics.ConnectionMetrics.ConnectionEstablished()
   at System.Net.Http.HttpConnectionBase..ctor(HttpConnectionPool pool, IPEndPoint remoteEndPoint)
   at System.Net.Http.HttpConnection..ctor(HttpConnectionPool pool, Stream stream, TransportContext transportContext, IPEndPoint remoteEndPoint)
   at System.Net.Http.HttpConnectionPool.ConstructHttp11ConnectionAsync(Boolean async, Stream stream, TransportContext transportContext, HttpRequestMessage request, IPEndPoint remoteEndPoint, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Net.Http.HttpConnectionPool.ConstructHttp11ConnectionAsync(Boolean async, Stream stream, TransportContext transportContext, HttpRequestMessage request, IPEndPoint remoteEndPoint, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Net.Security.SslStream.ReceiveHandshakeFrameAsync[TIOAdapter](CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.MoveNext()
   at System.Net.Sockets.SocketAsyncEventArgs.<>c.<.cctor>b__176_0(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
   at System.Threading.PortableThreadPool.IOCompletionPoller.Callback.Invoke(Event e)
   at System.Threading.ThreadPoolTypedWorkItemQueue`2.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

A similar exception but with a shorter stacktrace

[2023-08-29 12:11:38Z DBG] FirstChanceException event raised in InternalDynamics
System.ArgumentException: The list of label values must have the same number of elements as the list of label names.
   at Prometheus.LabelSequence..ctor(StringSequence names, StringSequence values)
[2023-08-29 12:11:38Z DBG]    at InternalDynamics.Util.DynasoftLogging.<>c.<LogFirstChanceExceptions>b__1_0(Object source, FirstChanceExceptionEventArgs e)
   at Prometheus.LabelSequence..ctor(StringSequence names, StringSequence values)
   at Prometheus.LabelSequence.From(StringSequence names, StringSequence values)
   at Prometheus.Collector`1.Labels(String[] labelValues)
   at Prometheus.Collector`1.WithLabels(String[] labelValues)
   at Prometheus.ManagedLifetimeMetricHandle`2.WithLease(Action`1 action, String[] labelValues)
   at Prometheus.MeterAdapter.OnMeasurementRecorded[T](Instrument instrument, T measurement, ReadOnlySpan`1 tags, Object state)
   at System.Diagnostics.Metrics.Instrument`1.RecordMeasurement(T measurement, ReadOnlySpan`1 tags)
   at System.Diagnostics.Metrics.Instrument`1.RecordMeasurement(T measurement, TagList& tagList)
   at System.Diagnostics.Metrics.UpDownCounter`1.Add(T delta, TagList& tagList)
   at Microsoft.AspNetCore.Hosting.HostingMetrics.RequestStart(Boolean isHttps, String scheme, String method, HostString host)
   at Microsoft.AspNetCore.Hosting.HostingApplicationDiagnostics.RecordRequestStartEventLog(HttpContext httpContext)
   at Microsoft.AspNetCore.Hosting.HostingApplication.CreateContext(IFeatureCollection contextFeatures)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.HandleRequest()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.HandleRequest()
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

[sandersaares]

This might be same as #452. Please try version 8.1.1 that incorporates a fix for this.