Cookie and BentoML Performance

[callmedada]

Hey Team,

I found an interesting thing and want to share with the team. Recently, when we perform the load test on BentoML 0.13 app, we found the cookie plays a significant role in the response time. For example, if we send a clean request without any cookie, the request take around 2 seconds (on average), and the transfer start is the most time consuming part.

However, if we keep the cookie and/or send within a session, it will be much faster (average 3x faster).

I am wondering if you have any clue regarding the reason behind? We do not want to use session because we are using the load balancer for only BentoML service. If we use the sticky session, the load balancing becoming meaningless, but if we do not use sticky session, I assume the performance will be still bad (because for example, if we use the roundrobin as the balancing rule, we will still need to go with the process to get a new cookie for every single request).

Any comments or criticizes are much appreciated.

Thanks so much,
Dennis

[callmedada]

Sorry, this issue seems unrelated. I end up solving this by not using encrypt transmission between pods inside the k8s cluster.