Add support for testing WebExtensions features

[queengooborg]

Automated tests for web extension features would help cover one of the biggest uncertainties in BCD (at least from my perspective). Unfortunately, web extensions are non-standard, so a feature list will have to be curated manually. Additionally, this requires writing a browser extension, which has its own set of complications.

@rebloor, you have almost single-handedly updated BCD's web extensions data. What are your thoughts on this? Would this be a huge benefit to your work?

[rebloor]

I'm not familiar with the automated test available for the BCD, so it's hard to judge the benefit. Superficially, having to maintain a manual list to validate a manually maintained list doesn't seem like it's a benefit.

[queengooborg]

In a nutshell, the collector project is designed to automatically update BCD's data by running tests in all the different browsers and versions (currently 2020+ releases of Chrome+Edge, Firefox and Safari for our goals) in both macOS and Windows. Using the results files that have been saved to its data repository, it then compiles BCD statements from the data, which are then used to update BCD itself. So for example, if our current data indicates that support was added for a feature in Chrome 85, but the collector finds that Chrome 85-86 have no support and Chrome 87+ does, it will update the data to state Chrome 87.

Besides just using the collector to update the data, I also want to use it as a way to validate the changes made. When it comes to web extensions, I've relied solely on your PRs and others' reviews to update the web extensions data, as I personally have no clue how to test and verify the data. The more that we can test with the collector, the more I can feel confident about your own updates, and the more I will feel confident about the accuracy of BCD.

[rebloor]

Thanks for the explanation. Considering the amount of effort involved, I would be more inclined to look at addressing concerns about the approval of web extensions BCD changes procedurally. For example, by allowing folks such as Rob or Luca to be added as reviewers and only merging once they have approved changes. As for historic inaccuracies, again it seems a considerable effort when it's unclear if we have a significant issue – I'm not aware of one, are you?

[queengooborg]

I am not aware of a specific existing issue with the data. However, I also was not aware about how many inaccuracies we had in our API, CSS and JS data either. Over the last year, I have been doing a lot of work toward synchronizing the collector results with BCD to fix inaccuracies, and while we were expecting this to be a quarterly goal, it has turned into a yearly goal instead, due to how many differences the collector still reports. What's more, it isn't just old wiki data, but recent PRs with data from credible (or, well, "credible" now) sources like ChromeStatus.com.

Unfortunately, we have no way of knowing how much of a payoff this will have. Before we can figure out how much of a payoff this will have for this category, we need to implement the tool for this category first. Simply based upon the fact that the API, CSS and JavaScript categories all had so, so many changes the collector's wanted to make, I think it's safe to say that it would be worth it to implement it for web extensions as well.

[rebloor]

I can understand the benefits of having such automated tests in place. My query is more about the logistics of getting it done, and I can't really comment on that. In the short term, I certainly think it would be helpful if web extensions related BCD PRs are not merged until they've had the OK from an appropriate SME. From my point of view, it is a little frustrating that I don't seem to be able to add them (e.g. Rob or Luca) to the PRs, but seem to have the ability to merge the PRs.