Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test runs used for scoring can sometimes revert to previous browser versions #188

Open
DanielRyanSmith opened this issue Sep 18, 2023 · 1 comment

Comments

@DanielRyanSmith
Copy link
Contributor

On September 14th, a test run of Chrome ran with labels "master" and "stable" for Chrome version 116. This came after a Chrome 117 was used for a run with the same labels on September 12th.

https://wpt.fyi/results/?diff&filter=ADC&run_id=5096725200502784&run_id=5118043404894208

This instance of running Chrome 117 as stable and then reverting to 116 resulted in a jump in the Chrome interop score for stable, then reverted that score.
interop-graph

This is likely a transient issue, as subsequent runs of stable have so far been using Chrome 117. If this becomes problematic in further scoring, or additional runs continue to fluctuate between using different browser versions, further investigation and action may be necessary.

cc @nairnandu

@gsnedders
Copy link
Member

Annoyingly, https://github.com/GoogleChromeLabs/chrome-for-testing/blob/gh-pages/known-good-versions-with-downloads.json doesn't have any history (presumably force-pushed each time). But given we simply download what's apparently a stable version, this is presumably a problem on the Chrome infra side?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants