fix: filter out crazy trials [DET-4782]

[hkang1]

Description

Extremely large metric values found in learning curve trials causes the uplot charts to break. Filter out any trials that have metric values that exceed 100,000 in value.

Test Plan

check to make sure experiment 725 on the latest-master loads ok when the metric is set to [Training] loss

Commentary (optional)

Checklist

• User-facing API changes need the "User-facing API Change" label.
• Release notes should be added as a separate file under docs/release-notes/.
  See Release Note for details.

[hamidzr]

does this come after #1796 ? this is deleting the experiment details pages

[hamidzr]

Code LGTM. Assuming users are not going to find real value in metric points with values outside the ALLOWED_METRIC_VLAUE range and won't really get confused with not seeing those here, this would be a good and easy improvement to make the charts more usable. cc @vishnu2kmohan

for a more complicated, less performant solution, employing outlier detection could be an option for the future 🤔

[vishnu2kmohan]

Assuming users are not going to find real value in metric points with values outside the ALLOWED_METRIC_VALUE range

Might we please log a console warning that we've encountered such an outlier (and also capture such instances via Telemetry, if it makes sense)?

I'm still a little concerned that there may arise a situation where a metric may genuinely lie outside and that we are dropping it due to an implementation constraint.

cc @ahj-determined

[hkang1]

Might we please log a console warning that we've encountered such an outlier (and also capture such instances via Telemetry, if it makes sense)?

I'm still a little concerned that there may arise a situation where a metric may genuinely lie outside and that we are dropping it due to an implementation constraint.

@vishnu2kmohan this is only the reporting end when displaying the learning curve. We're not actually dropping metric values from the experiment. If we want to log such a case, it would make sense for the backend to capture this when it encounters it there. uPlot does not render numbers in the billions either, and talking with @katport, if the metric values are exceeding such a high value, it's more likely that something went wrong.

[hkang1]

does this come after #1796 ? this is deleting the experiment details pages

rebased to remove all those intersecting commits

[leeoniya]

uPlot does not render numbers in the billions either

this sounds odd; it can certainly render values that high: https://jsfiddle.net/0mvdktqf/

if you want to squash spikes, you can tweak y scale auto-ranging (e.g. set soft max / hard max): https://leeoniya.github.io/uPlot/demos/soft-minmax.html

or use log scales:
https://leeoniya.github.io/uPlot/demos/log-scales2.html

might also want to check out these recent discussions:

grafana/grafana#979
grafana/grafana#30326

cheers!

[hkang1]

uPlot does not render numbers in the billions either

this sounds odd; it can certainly render values that high: https://jsfiddle.net/0mvdktqf/

if you want to squash spikes, you can tweak y scale auto-ranging (e.g. set soft max / hard max): https://leeoniya.github.io/uPlot/demos/soft-minmax.html

or use log scales:
https://leeoniya.github.io/uPlot/demos/log-scales2.html

might also want to check out these recent discussions:

grafana/grafana#979
grafana/grafana#30326

cheers!

Hey @leeoniya thanks for the comment and the suggestions! Looks like it rendered just fine with the single digits and the billion+ values. I'll try to nail down exactly what the issue is, but introduction of time series with these giant numbers among "normal" series with values between 0 ~ 10 is triggering a blank chart without an error.