You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we use the CI output to check whether parameters are changing, and if they are changing by a small amount (say less than 1%), we typically say that is ok, merge the PR, then issue a new tagged release
What we are missing is a way to ensure we're not drifting. Say the number of hits in the OD changes by +0.5% in 5 different PRs that are separated from each other by releases. What we see in the CI is a +0.5% change, which is valid, so we accept it. What we miss is that the number of OD hits has increased by 2.5% over the course of these PRs/releases.
To account for this, we need to do something like comparing new reference files with multiple previous reference files (e.g. from previous release, release -5, release -10, etc.)
Note that this comparison gets less trivial the further you go back, as config options can & will change
The text was updated successfully, but these errors were encountered:
Currently, we use the CI output to check whether parameters are changing, and if they are changing by a small amount (say less than 1%), we typically say that is ok, merge the PR, then issue a new tagged release
What we are missing is a way to ensure we're not drifting. Say the number of hits in the OD changes by +0.5% in 5 different PRs that are separated from each other by releases. What we see in the CI is a +0.5% change, which is valid, so we accept it. What we miss is that the number of OD hits has increased by 2.5% over the course of these PRs/releases.
To account for this, we need to do something like comparing new reference files with multiple previous reference files (e.g. from previous release, release -5, release -10, etc.)
Note that this comparison gets less trivial the further you go back, as config options can & will change
The text was updated successfully, but these errors were encountered: