[K8S] HPA working but loadbalancing not working as expected

[Ronan-WeScale]

I have HPA enabled on my cortex-tenant deployment, but after scale, other pod doesn't receive timeseries.
When I check"cortex_tenant_timeseries_received" metrics only the first pod has count
But when I check cortex_tenant_timeseries_batches_received metrics, other pod receive some batch but no much like the first pod :

Result series: 3

pod 1 : 25353
pod 2: 4
pod 3 : 15

Maybe we can add an nginx proxy (or other app) to loadbalance the query between the backend ?
Tell me and I can try to implement it.

[Ronan-WeScale]

maybe the problem come from prometheus remote write process, I need to analyse this

[ksrt12]

maybe the problem come from prometheus remote write process

I did some research and the problem is indeed prometheus shards (see queue_config).
Prometheus uses an auto-sizing algorithm for the number of shards, and it's entirely possible that the number of shards is 1, so only 1 cortex-tenant pod is used.