Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document the data collection and results generation system #357

Open
jgraham opened this issue Jun 9, 2023 · 2 comments · May be fixed by #667
Open

Document the data collection and results generation system #357

jgraham opened this issue Jun 9, 2023 · 2 comments · May be fixed by #667

Comments

@jgraham
Copy link
Contributor

jgraham commented Jun 9, 2023

The way that we go from tests to results to data on wpt.fyi to interop scores is under documented an unclear. There are two specific problems here (which maybe require different solutions, but it's worth considering as part of one higher level issue):

  • People don't know what to expect e.g. in terms of latency, and don't know at what point to raise an alarm when something isn't working the way they expect (e.g. when browser changes aren't showing up on the interop dashboard).
  • It's unclear what parts of the pipeline are considered stable and can be relied on by external tooling (e.g. for vendors wanting to understand how specific in-progress changes affect interop).

That suggests that for 2024 we should consider two improvements:

  • Produce documentation of the overall dataflow, so that there's a reference to point people to for all the systems involved, what they should expect in terms of latency, and what to do when something doesn't work in the way they expect.
  • Provide a stable API for accessing the data and tooling needed to recreate the interop scores. Currently there are a variety of data files in different, undocumented, formats, that one must download from different systems, which are quite specialised to the needs of wpt.fyi. That makes external tooling both difficult to create in the first place, and liable to break whenever the needs of wpt.fyi change.
@bkardell
Copy link
Contributor

We agree that these are good/important things to put on our collective priorities list. That said, other than offering to help editorially with documentation or in promoting/discussing documentation when it arrives, there probably isn't a lot that @meyerweb and I can offer in terms of driving those efforts.

@foolip
Copy link
Member

foolip commented May 31, 2024

I have sent #667. Editorial review is the main thing that's needed.

@foolip foolip linked a pull request May 31, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants