Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Surprising results when scoring interop when a labelled test no longer exists #149

Open
gsnedders opened this issue Feb 9, 2023 · 0 comments

Comments

@gsnedders
Copy link
Member

From web-platform-tests/interop#281, we hit a case where a number of scores moved away from 100%, which people weren't expecting.

This is because of:

// We always normalize against the number of tests we are looking for,
// rather than the total number of tests we found. The trade-off is all
// about new tests being added to the set.

and web-platform-tests/wpt#38413 splitting this test up into variants, thus making the old test URL no longer a test URL. It is, however, still labelled.

We should, at the very least, track what tests in allTestsSet we've seen, and log those that we didn't find a result for. This would put something in the log and make it much easier to understand what's happening. Though perhaps with older runs and labelling changes this would be too noisy?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant