Glimpse hangs in GlimpseRuntime.GetTabStore

[anton-gogolev]

I'll try to be as coherent as possible here.

We are using Glimpse (Glimpse 1.8.6, Glimpse.Ado 1.7.3, Glimpse.AspNet 1.8.1, Glimpse.EF6 1.6.5, Glimpse.Mvc5 1.5.3) in an ASP.NET MVC 5 application and occasionally experience high CPU usage -- basically, out of nowhere.

Here is what my WinDbg-fu has shown so far.

There are three runaway threads with huge run-times:

 User Mode Time
  Thread       Time
  43:1bf8      0 days 0:36:36.375
  42:2144      0 days 0:36:28.703
  40:cd4       0 days 0:35:38.390
  31:23b0      0 days 0:01:29.859

Examining call stacks, all of them are stuck in about the same place:

Thread  42
Current frame: (MethodDesc 00007ffa8ca64e38 +0x93 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].FindEntry(System.__Canon))
Child-SP         RetAddr          Caller, Callee
0000003d2f99b100 00007ffa8ce4a755 (MethodDesc 00007ffa8ca6a808 +0xf5 System.Collections.Hashtable.get_Item(System.Object))
0000003d2f99b110 00007ffa8ce4e5e5 (MethodDesc 00007ffa8ca6a7b8 +0xa5 System.Collections.Hashtable.ContainsKey(System.Object))
0000003d2f99b160 00007ffa8ce50c6a (MethodDesc 00007ffa8ca64df8 +0xa System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].ContainsKey(System.__Canon)), calling (MethodDesc 00007ffa8ca64e38 +0 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].FindEntry(System.__Canon))
0000003d2f99b190 00007ffa3162baa0 (MethodDesc 00007ffa30969430 +0xe0 Glimpse.Core.Framework.GlimpseRuntime.GetTabStore(System.String))
0000003d2f99b1a0 00007ffa8e1c625f clr!SystemNative::__GetSystemTimeAsFileTime+0xf, calling kernel32!GetSystemTimeAsFileTimeStub
0000003d2f99b1e0 00007ffa3162b867 (MethodDesc 00007ffa317a1468 +0x27 Glimpse.Core.Extensions.TabContextExtensions.PersistMessage[[System.__Canon, mscorlib]](System.__Canon, Glimpse.Core.Extensibility.ITabSetupContext))
0000003d2f99b1f0 00007ffa8ce211a1 (MethodDesc 00007ffa8ca5d598 +0xc1 System.Collections.Generic.Dictionary`2+Enumerator[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext()), calling clr!JIT_CheckedWriteBarrier
0000003d2f99b200 00007ffa3162b535 (MethodDesc 00007ffa30e450c0 +0x35 Glimpse.Core.Extensibility.Subscriber`1[[System.__Canon, mscorlib]].Execute(System.Object)), calling clr!JIT_ChkCastAny
0000003d2f99b240 00007ffa30ca86c6 (MethodDesc 00007ffa30e90390 +0xe6 Glimpse.Core.Extensibility.MessageBroker.Publish[[System.__Canon, mscorlib]](System.__Canon))
0000003d2f99b260 00007ffa3162b43d (MethodDesc 00007ffa306c7b28 +0x4d Glimpse.Ado.Message.AdoMessage..ctor(System.Guid)), calling (MethodDesc 00007ffa8ca79720 +0 System.DateTime.get_Now())
0000003d2f99b270 00007ffa3162b176 (MethodDesc 00007ffa30070ca0 +0x3c6 Glimpse.Ado.AlternateType.Support.LogCommandStart(Glimpse.Ado.AlternateType.GlimpseDbCommand, System.Guid, System.TimeSpan, Boolean))
0000003d2f99b2d0 00007ffa3162b123 (MethodDesc 00007ffa30070ca0 +0x373 Glimpse.Ado.AlternateType.Support.LogCommandStart(Glimpse.Ado.AlternateType.GlimpseDbCommand, System.Guid, System.TimeSpan, Boolean))
0000003d2f99b300 00007ffa8cef9ec8 (MethodDesc 00007ffa8ca82170 +0x78 DomainNeutralILStubClass.IL_STUB_PInvoke(System.Guid ByRef))
0000003d2f99b3c0 00007ffa3162ab78 (MethodDesc 00007ffa306c63b8 +0x68 Glimpse.Ado.AlternateType.GlimpseDbCommand.ExecuteDbDataReader(System.Data.CommandBehavior)), calling (MethodDesc 00007ffa30070ca0 +0 Glimpse.Ado.AlternateType.Support.LogCommandStart(Glimpse.Ado.AlternateType.GlimpseDbCommand, System.Guid, System.TimeSpan, Boolean))
0000003d2f99b400 00007ffa8cec31c4 (MethodDesc 00007ffa8ca7c760 +0x204 System.Collections.Generic.List`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>)), calling (MethodDesc 00007ffa8ca7c760 +0x21d System.Collections.Generic.List`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>))
0000003d2f99b490 00007ffa311e7d52 (MethodDesc 00007ffa31340778 +0x62 System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1[[System.__Canon, mscorlib]].Dispatch[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.__Canon, System.Func`3<System.__Canon,System.__Canon,System.__Canon>, System.__Canon, System.Action`3<System.__Canon,System.__Canon,System.__Canon>, System.Action`3<System.__Canon,System.__Canon,System.__Canon>))
0000003d2f99b4b0 00007ffa311c9ba7 (MethodDesc 00007ffa303db0a0 +0x167 System.Data.Entity.Infrastructure.Interception.DbInterceptionContext..ctor(System.Data.Entity.Infrastructure.Interception.DbInterceptionContext)), calling clr!JIT_WriteBarrier
0000003d2f99b540 00007ffa3162aaa9 (MethodDesc 00007ffa303df9c8 +0x1f9 System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(System.Data.Common.DbCommand, System.Data.Entity.Infrastructure.Interception.DbCommandInterceptionContext)), calling (MethodDesc 00007ffa31340778 +0 System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1[[System.__Canon, mscorlib]].Dispatch[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.__Canon, System.Func`3<System.__Canon,System.__Canon,System.__Canon>, System.__Canon, System.Action`3<System.__Canon,System.__Canon,System.__Canon>, System.Action`3<System.__Canon,System.__Canon,System.__Canon>))
0000003d2f99b5b0 00007ffa31627904 (MethodDesc 00007ffa30480538 +0x24 System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(System.Data.Entity.Core.EntityClient.EntityCommand, System.Data.CommandBehavior)), calling 00007ffa311653e0
0000003d2f99b600 00007ffa316260c1 (MethodDesc 00007ffa314c0e08 +0x91 System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlan.Execute[[System.__Canon, mscorlib]](System.Data.Entity.Core.Objects.ObjectContext, System.Data.Entity.Core.Objects.ObjectParameterCollection))
0000003d2f99b690 00007ffa3124a193 (MethodDesc 00007ffa314c0300 +0x103 System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[[System.__Canon, mscorlib]](System.Func`1<System.__Canon>, System.Data.Entity.Infrastructure.IDbExecutionStrategy, Boolean, Boolean))
0000003d2f99b6b0 00007ffa311c9734 (MethodDesc 00007ffa30073b60 +0x14 System.Data.Entity.Utilities.Check.NotEmpty(System.String, System.String)), calling (MethodDesc 00007ffa8ca60608 +0 System.String.IsNullOrWhiteSpace(System.String))
0000003d2f99b700 00007ffa8e1c1f2c clr!InstantiatingMethodStubWorker+0xbc
0000003d2f99b710 00007ffa8e1c1ed8 clr!InstantiatingMethodStubWorker+0x68, calling clr!GetThread
0000003d2f99b7b0 00007ffa31249f8b (MethodDesc 00007ffa3144f200 +0xcb System.Data.Entity.Core.Objects.ObjectQuery`1+<>c__DisplayClass7[[System.__Canon, mscorlib]].<GetResults>b__5())
0000003d2f99b820 00007ffa31249df7 (MethodDesc 00007ffa3144f878 +0x57 System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[[System.__Canon, mscorlib]](System.Func`1<System.__Canon>))
0000003d2f99b830 00007ffa8e267cda clr!JIT_VirtualFunctionPointer+0x9a, calling clr!JitGenericHandleCacheTraits::CompareKeys
0000003d2f99b850 00007ffa31249cb3 (MethodDesc 00007ffa30c24c48 +0x13 System.Data.Entity.SqlServer.SqlProviderServices.<.ctor>b__0()), calling clr!JIT_TrialAllocSFastMP_InlineGetThread
0000003d2f99b880 00007ffa31249872 (MethodDesc 00007ffa3141a2a8 +0x162 System.Data.Entity.Core.Objects.ObjectQuery`1[[System.__Canon, mscorlib]].GetResults(System.Nullable`1<System.Data.Entity.Core.Objects.MergeOption>))
0000003d2f99b8e0 00007ffa312496ee (MethodDesc 00007ffa3141a2c0 +0x1e System.Data.Entity.Core.Objects.ObjectQuery`1[[System.__Canon, mscorlib]].<System.Collections.Generic.IEnumerable<T>.GetEnumerator>b__0()), calling (MethodDesc 00007ffa3141a2a8 +0 System.Data.Entity.Core.Objects.ObjectQuery`1[[System.__Canon, mscorlib]].GetResults(System.Nullable`1<System.Data.Entity.Core.Objects.MergeOption>))
0000003d2f99b910 00007ffa31249644 (MethodDesc 00007ffa3144e3e8 +0x24 System.Data.Entity.Internal.LazyEnumerator`1[[System.__Canon, mscorlib]].MoveNext())
0000003d2f99b920 00007ffa312444ba (MethodDesc 00007ffa31448020 +0x4a System.Data.Entity.Core.Objects.ELinq.ObjectQueryProvider.CreateQuery[[System.__Canon, mscorlib]](System.Linq.Expressions.Expression)), calling (MethodDesc 00007ffa304e64d0 +0 System.Data.Entity.Core.Objects.ELinq.ObjectQueryProvider.GetObjectQueryState(System.Data.Entity.Core.Objects.ObjectQuery, System.Linq.Expressions.Expression, System.Type))
0000003d2f99b960 00007ffa8b678f12 (MethodDesc 00007ffa8b410698 +0x92 System.Linq.Enumerable.FirstOrDefault[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>))
0000003d2f99b9d0 00007ffa31990992 (MethodDesc 00007ffa301c83c0 +0x382 REDACTED.DAL.REDACTEDSettings.REDACTEDSettingsRepository.IsFeatureEnabled(System.String, System.Nullable`1<Int32>, System.Nullable`1<Int32>, System.Nullable`1<Int32>)), calling (MethodDesc 00007ffa8b4f9eb8 +0 System.Linq.Queryable.FirstOrDefault[[System.__Canon, mscorlib]](System.Linq.IQueryable`1<System.__Canon>, System.Linq.Expressions.Expression`1<System.Func`2<System.__Canon,Boolean>>))
0000003d2f99c0a0 00007ffa319d63ac (MethodDesc 00007ffa301b6e10 +0xfc REDACTED.Infrastructure.REDACTEDFeatures.IsFeatureEnabledFor(REDACTED.Core.Company, System.String, System.Nullable`1<Int32>)), calling (MethodDesc 00007ffa301c83c0 +0 REDACTED.DAL.REDACTEDSettings.REDACTEDSettingsRepository.IsFeatureEnabled(System.String, System.Nullable`1<Int32>, System.Nullable`1<Int32>, System.Nullable`1<Int32>))
0000003d2f99c150 00007ffa319d74d2 (MethodDesc 00007ffa301b6e00 +0x52 REDACTED.Infrastructure.REDACTEDFeatures.CalculateRackPrice(REDACTED.Core.Company)), calling (MethodDesc 00007ffa301b6e10 +0 REDACTED.Infrastructure.REDACTEDFeatures.IsFeatureEnabledFor(REDACTED.Core.Company, System.String, System.Nullable`1<Int32>))
0000003d2f99c1a0 00007ffa319d7010 (MethodDesc 00007ffa301b91b0 +0x270 REDACTED.Infrastructure.WSCommonEndpoint.GetPricingRequest(Int32, System.Collections.Generic.List`1<System.Guid>, System.DateTime, System.DateTime, Int32, REDACTED.Core.Company, System.Nullable`1<Int32>, System.Nullable`1<REDACTED.Core.ApplicationType>, REDACTED.Core.Language, Boolean, REDACTED.Core.PlaceOfPayment, Boolean, System.String, System.String)), calling (MethodDesc 00007ffa301b6e00 +0 REDACTED.Infrastructure.REDACTEDFeatures.CalculateRackPrice(REDACTED.Core.Company))
0000003d2f99c2a0 00007ffa319d6533 (MethodDesc 00007ffa301b91c0 +0xf3 REDACTED.Infrastructure.WSCommonEndpoint.GetHotelPrices(Int32, System.Collections.Generic.List`1<System.Guid>, System.DateTime, System.DateTime, Int32, REDACTED.Core.Company, System.Nullable`1<Int32>, System.Nullable`1<REDACTED.Core.ApplicationType>, REDACTED.Core.Language, Boolean, REDACTED.Core.PlaceOfPayment, Boolean, System.String, System.String)), calling (MethodDesc 00007ffa301b91b0 +0 REDACTED.Infrastructure.WSCommonEndpoint.GetPricingRequest(Int32, System.Collections.Generic.List`1<System.Guid>, System.DateTime, System.DateTime, Int32, REDACTED.Core.Company, System.Nullable`1<Int32>, System.Nullable`1<REDACTED.Core.ApplicationType>, REDACTED.Core.Language, Boolean, REDACTED.Core.PlaceOfPayment, Boolean, System.String, System.String))
0000003d2f99c5b0 00007ffa323912ec (MethodDesc 00007ffa301d24a8 +0x1fc REDACTED.Infrastructure.WSCommonEndpoint+<>c__DisplayClass13_0.<GetAlternateRatesForService>b__0(System.Guid)), calling (MethodDesc 00007ffa301b91c0 +0 REDACTED.Infrastructure.WSCommonEndpoint.GetHotelPrices(Int32, System.Collections.Generic.List`1<System.Guid>, System.DateTime, System.DateTime, Int32, REDACTED.Core.Company, System.Nullable`1<Int32>, System.Nullable`1<REDACTED.Core.ApplicationType>, REDACTED.Core.Language, Boolean, REDACTED.Core.PlaceOfPayment, Boolean, System.String, System.String))
0000003d2f99c840 00007ffa2fbedfdc (MethodDesc 00007ffa321ecc60 +0x3c System.Threading.Tasks.Parallel+<>c__DisplayClass31_0`2[[System.Guid, mscorlib],[System.__Canon, mscorlib]].<ForEachWorker>b__0(Int32))
0000003d2f99c860 00007ffa8e1c653a clr!SystemNative::GetTickCount+0xa, calling kernel32!GetTickCountStub
0000003d2f99c890 00007ffa8d83c28e (MethodDesc 00007ffa8cca2138 +0x28e System.Threading.Tasks.Parallel+<>c__DisplayClass17_0`1[[System.__Canon, mscorlib]].<ForWorker>b__1())
0000003d2f99c8b0 00007ffa8ce51673 (MethodDesc 00007ffa8ca509a0 +0x23 System.Threading.ThreadPool.UnsafeQueueCustomWorkItem(System.Threading.IThreadPoolWorkItem, Boolean)), calling (MethodDesc 00007ffa8ca509a0 +0x2a System.Threading.ThreadPool.UnsafeQueueCustomWorkItem(System.Threading.IThreadPoolWorkItem, Boolean))
0000003d2f99c950 00007ffa8d82b430 (MethodDesc 00007ffa8ca75878 +0x20 System.Threading.Tasks.Task.InnerInvokeWithArg(System.Threading.Tasks.Task))
0000003d2f99c980 00007ffa8d902324 (MethodDesc 00007ffa8cb98550 +0x114 System.Threading.Tasks.Task+<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(System.Object)), calling (MethodDesc 00007ffa8ca75878 +0 System.Threading.Tasks.Task.InnerInvokeWithArg(System.Threading.Tasks.Task))
0000003d2f99c9f0 00007ffa8ce4ca72 (MethodDesc 00007ffa8ca7a660 +0x162 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
0000003d2f99cac0 00007ffa8ce4c904 (MethodDesc 00007ffa8ca7a650 +0x14 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)), calling (MethodDesc 00007ffa8ca7a660 +0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
0000003d2f99caf0 00007ffa8ce9d6dc (MethodDesc 00007ffa8ca75850 +0x21c System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)), calling (MethodDesc 00007ffa8ca7a650 +0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
0000003d2f99cb10 00007ffa8cec17be (MethodDesc 00007ffa8ca60688 +0x1e System.String.GetHashCode()), calling clr!JIT_GetSharedNonGCStaticBase_InlineGetAppDomain
0000003d2f99cb50 00007ffa8ce13875 (MethodDesc 00007ffa8ca6a788 +0x15 System.Collections.Hashtable.InitHash(System.Object, Int32, UInt32 ByRef, UInt32 ByRef))
0000003d2f99cba0 00007ffa8ce9cdf3 (MethodDesc 00007ffa8ca75840 +0x73 System.Threading.Tasks.Task.ExecuteEntry(Boolean)), calling (MethodDesc 00007ffa8ca75850 +0 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef))
0000003d2f99cbe0 00007ffa8cea1293 (MethodDesc 00007ffa8cbbaa30 +0x33 System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)), calling (MethodDesc 00007ffa8ca75840 +0 System.Threading.Tasks.Task.ExecuteEntry(Boolean))
0000003d2f99cbf0 00007ffa8ce9c2e5 (MethodDesc 00007ffa8ca75500 +0x15 System.Threading.Tasks.Task.get_CurrentStackGuard()), calling clr!JIT_GetSharedGCThreadStaticBase
0000003d2f99cc30 00007ffa8cea1151 (MethodDesc 00007ffa8cbd2bf0 +0x1a1 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
0000003d2f99cc78 00007ffa8e337e2c clr!JIT_NewArr1+0x7c, calling clr!LazyMachStateCaptureState
0000003d2f99ccd0 00007ffa8d82a669 (MethodDesc 00007ffa8ca754a0 +0xa9 System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler, Boolean)), calling (MethodDesc 00007ffa8cbd2bf0 +0 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
0000003d2f99cd60 00007ffa8d8377b1 (MethodDesc 00007ffa8cc4fc48 +0x401 System.Threading.Tasks.Parallel.ForWorker[[System.__Canon, mscorlib]](Int32, Int32, System.Threading.Tasks.ParallelOptions, System.Action`1<Int32>, System.Action`2<Int32,System.Threading.Tasks.ParallelLoopState>, System.Func`4<Int32,System.Threading.Tasks.ParallelLoopState,System.__Canon,System.__Canon>, System.Func`1<System.__Canon>, System.Action`1<System.__Canon>)), calling (MethodDesc 00007ffa8ca75490 +0 System.Threading.Tasks.Task.RunSynchronously(System.Threading.Tasks.TaskScheduler))
0000003d2f99cd70 00007ffa8cef0dcd (MethodDesc 00007ffa8cbbc370 +0x2d System.Collections.Generic.List`1[[System.Guid, mscorlib]].EnsureCapacity(Int32)), calling (MethodDesc 00007ffa8cbbc288 +0 System.Collections.Generic.List`1[[System.Guid, mscorlib]].set_Capacity(Int32))
0000003d2f99cd90 00007ffa2fbedc4e (MethodDesc 00007ffa321eb1a8 +0xee System.Threading.Tasks.Parallel.ForEachWorker[[System.Guid, mscorlib],[System.__Canon, mscorlib]](System.Collections.Generic.IList`1<System.Guid>, System.Threading.Tasks.ParallelOptions, System.Action`1<System.Guid>, System.Action`2<System.Guid,System.Threading.Tasks.ParallelLoopState>, System.Action`3<System.Guid,System.Threading.Tasks.ParallelLoopState,Int64>, System.Func`4<System.Guid,System.Threading.Tasks.ParallelLoopState,System.__Canon,System.__Canon>, System.Func`5<System.Guid,System.Threading.Tasks.ParallelLoopState,Int64,System.__Canon,System.__Canon>, System.Func`1<System.__Canon>, System.Action`1<System.__Canon>)), calling clr!JIT_WriteBarrier
0000003d2f99ce20 00007ffa2fbed657 (MethodDesc 00007ffa321e8ad0 +0x167 System.Threading.Tasks.Parallel.ForEachWorker[[System.Guid, mscorlib],[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.Guid>, System.Threading.Tasks.ParallelOptions, System.Action`1<System.Guid>, System.Action`2<System.Guid,System.Threading.Tasks.ParallelLoopState>, System.Action`3<System.Guid,System.Threading.Tasks.ParallelLoopState,Int64>, System.Func`4<System.Guid,System.Threading.Tasks.ParallelLoopState,System.__Canon,System.__Canon>, System.Func`5<System.Guid,System.Threading.Tasks.ParallelLoopState,Int64,System.__Canon,System.__Canon>, System.Func`1<System.__Canon>, System.Action`1<System.__Canon>)), calling (MethodDesc 00007ffa321eb1a8 +0 System.Threading.Tasks.Parallel.ForEachWorker[[System.Guid, mscorlib],[System.__Canon, mscorlib]](System.Collections.Generic.IList`1<System.Guid>, System.Threading.Tasks.ParallelOptions, System.Action`1<System.Guid>, System.Action`2<System.Guid,System.Threading.Tasks.ParallelLoopState>, System.Action`3<System.Guid,System.Threading.Tasks.ParallelLoopState,Int64>, System.Func`4<System.Guid,System.Threading.Tasks.ParallelLoopState,System.__Canon,System.__Canon>, System.Func`5<System.Guid,System.Threading.Tasks.ParallelLoopState,Int64,System.__Canon,System.__Canon>, System.Func`1<System.__Canon>, System.Action`1<System.__Canon>))
0000003d2f99cee0 00007ffa2fbed044 (MethodDesc 00007ffa321e87b8 +0x84 System.Threading.Tasks.Parallel.ForEach[[System.Guid, mscorlib]](System.Collections.Generic.IEnumerable`1<System.Guid>, System.Action`1<System.Guid>)), calling (MethodDesc 00007ffa321e8ad0 +0 System.Threading.Tasks.Parallel.ForEachWorker[[System.Guid, mscorlib],[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.Guid>, System.Threading.Tasks.ParallelOptions, System.Action`1<System.Guid>, System.Action`2<System.Guid,System.Threading.Tasks.ParallelLoopState>, System.Action`3<System.Guid,System.Threading.Tasks.ParallelLoopState,Int64>, System.Func`4<System.Guid,System.Threading.Tasks.ParallelLoopState,System.__Canon,System.__Canon>, System.Func`5<System.Guid,System.Threading.Tasks.ParallelLoopState,Int64,System.__Canon,System.__Canon>, System.Func`1<System.__Canon>, System.Action`1<System.__Canon>))
0000003d2f99cf60 00007ffa32390ee6 (MethodDesc 00007ffa301b91d0 +0x456 REDACTED.Infrastructure.WSCommonEndpoint.GetAlternateRatesForService(REDACTED.Models.Application.Accommodation)), calling (MethodDesc 00007ffa321e87b8 +0 System.Threading.Tasks.Parallel.ForEach[[System.Guid, mscorlib]](System.Collections.Generic.IEnumerable`1<System.Guid>, System.Action`1<System.Guid>))

According to Tess Fernandez, this is how concurrent access to a Dictionary<> instance manifests itself.

Here is what I think is happening. When the GetAlternateRatesForService() methods spins up a Parallel.ForEach, each "parallel worker" threads receives an instance of HttpContext hidden in a Logicall Call Context:

private static HttpContextBase TryGetOrCaptureLogicalContext()
{
    if (AsyncSupportDisabled)
    {
        return new HttpContextWrapper(HttpContext.Current);
    }

    if (HttpContext.Current == null)
        return AntiSerializationWrapper<HttpContextBase>.Unwrap(CallContext.LogicalGetData("Glimpse.HttpContext"));

    var context = new HttpContextWrapper(HttpContext.Current);
    CallContext.LogicalSetData("Glimpse.HttpContext", new AntiSerializationWrapper<HttpContextBase>(context));

    return context;
}

The context instance is now effectively shared between "parallel worker" threads, and it is now a matter of time for GlimpseRuntime.GetTabStore() to be accessed concurrently, which in turn corrupts something inside HttpContext.Items dictionary.

Does this sound plausible?