Timeout awaiting response issue.

[avolt]

Hey team!

I am using google cloud.

Sometimes an error occurs when there are a lot of connection requests.

cpu is at 30%
Redis usage wasn't much.

Can you help me with this?

I was a node js developer and I don't know much about the .net core ecosystem.

I'm sorry I'm not good at English.

[ERROR]|Microsoft.AspNetCore.Server.Kestrel|Connection id "0HN23RLK7FKE4",
Request id "0HN23RLK7FKE4:000000E0": An unhandled exception was thrown by the application.
StackExchange.Redis.RedisTimeoutException: Timeout awaiting response
(outbound=26KiB, inbound=38KiB, 5016ms elapsed, timeout is 5000ms),
command=EXPIRE,
next: HGETALL session:some:info:10108995606,
inst: 0, qu: 166, qs: 446, aw: True, bw: CheckingForTimeoutComplete,
rs: DequeueResult, ws: Writing, in: 0, in-pipe: 77194, out-pipe: 0,
last-in: 0, cur-in: 35572, sync-ops: 0, async-ops: 38340728,
serverEndpoint: x.x.x.x:6379,
conn-sec: 71086.43, aoc: 0, mc: 1/1/0,
mgr: 9 of 10 available,
clientName: dev01, IOCP: (Busy=0,Free=1000,Min=500,Max=1000),
WORKER: (Busy=196,Free=32571,Min=400,Max=32767),
POOL: (Threads=420,QueuedItems=57,CompletedItems=486767659,Timers=58),
v: 2.7.33.41805
(Please take a look at this article for some common client-side
issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at alchemist_server.Manager.RedisManager.KeyExpire(REDIS_DB_TYPE redisType, String key, Int32 expireSec)
in /src/Manager/RedisManager.cs:line 271
at alchemist_server.Service.SessionService.SessionValidation(HttpContext context)
in /src/Service/SessionService.cs:line 361
at alchemist_server.Middleware.SessionVerification.InvokeAsync(HttpContext context,
IRedisManager redisManager, ISessionService sessionService)
in /src/Middleware/SessionVerification.cs:line 47
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests
[TContext](IHttpApplication`1 application)

[NickCraver]

The problem you're having is thread pool starvation somewhere, indicated by:

WORKER: (Busy=196,Free=32571,Min=400,Max=32767),
POOL: (Threads=420,QueuedItems=57,CompletedItems=486767659,Timers=58),

I see you're all async against Redis so this is likely somewhere else in the application. I'd suggest taking a memory dump under load to see what all those threads are doing.