High outliers for some REST API responses

[nflaig]

Based on the metrics, it looks like some REST API responses have pretty high outliers while the average responses times are quite low.

Requests pulling data from network

Might be related to the API pulling (or pushing) data from the network

getPeers

(needs more buckets between 100ms-1s)

prepareBeaconCommitteeSubnet

(needs more buckets between 100ms-1s)

Requests pulling data from state

While other APIs that involve just pulling data from head state don't have this issue

getStateFinalityCheckpoints

getStateValidators

(querying for low validator count)

[g11tech]

one should measure 95 percentile response, as thats what other seems to be measuring

[nflaig]

one should measure 95 percentile response, as thats what other seems to be measuring

That's a good idea to have as a panel as avg time might not be a good indicator a lot of the times. Histograms (by bucket) is the best way to detect high outliers though, but the problem is that it does not work well with api responses times as we need to look at the data per api (operationId) so we you would need a panel per api to properly visualize it.

But the fact that there are some getPeers requests that take > 1 second is definitely something we need to investigate.

Looking at the 95th percentile for getPeers it still looks fine, we might need more buckets between 100ms-1s to get better data.