Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Karma is deprecated #5547

Open
joeyparrish opened this issue Aug 28, 2023 · 3 comments
Open

Karma is deprecated #5547

joeyparrish opened this issue Aug 28, 2023 · 3 comments
Labels
priority: P3 Useful but not urgent type: CI An issue with our continuous integration tests
Milestone

Comments

@joeyparrish
Copy link
Member

I just found out that Karma was officially deprecated 4 months ago. https://dev.to/this-is-angular/angular-testing-in-2023-past-present-and-future-j5m

This isn't urgent, but we should decide what our test runner will look like in the future.

Things we need:

  1. Real browser testing through Selenium grid so we can test video playback on real devices (not using jsdom or phantomjs)
  2. Testing in-browser through a URL, instead of through WebDriver commands that manipulate the browser (this gets us the exotic devices in our lab through generic-webdriver-server)
  3. Serve tests over HTTPS with a custom certificate (without which we lose EME coverage and encrypted playback tests in the lab)
  4. A way for the in-browser tests to invoke the WebDriver screenshot command in the test runner and have the runner compare that screenshot to the reference screenshots (today we have a custom Karma middleware that the tests can call into)
  5. Machine-readable test coverage reports

Things that would be make the transition easier:

  1. Support for our current test framework, Jasmine (though we could rewrite our tests if necessary)
  2. Test coverage output in the same JSON format we parse today in our workflows (though we could hypothetically convert to that or rewrite our workflows)
  3. A way to run local browser tests over WebDriver, maybe with a plugin (this is how we get screenshot tests without using the lab today, via karma-local-wd-launcher)

Input from the community is very welcome. I've done very little research so far.

@joeyparrish joeyparrish added priority: P3 Useful but not urgent type: CI An issue with our continuous integration tests labels Aug 28, 2023
@joeyparrish joeyparrish pinned this issue Aug 28, 2023
@github-actions github-actions bot added this to the Backlog milestone Aug 28, 2023
@joeyparrish
Copy link
Member Author

joeyparrish commented Oct 4, 2023

I believe jasmine-browser-runner is the way forward. It will let us keep our existing Jasmine tests, it will connect to a remote Selenium grid, and we can reimplement our custom middlewares in express via configuration.

I'm not yet sure about:

  1. TLS configuration for testing over HTTPS Resolved by my PR, Add TLS and hostname support jasmine/jasmine-browser-runner#49
  2. Launching local WebDriver instances to support local screenshot testing (probably could do it by starting with something like this)
  3. Coverage reporting

joeyparrish added a commit to joeyparrish/jasmine-browser-runner that referenced this issue Mar 8, 2024
This adds both TLS and hostname support.

TLS support is activated by specifying paths to a key and a cert.
There are caveats about self-signed certs in the README.

This also adds hostname support. This impacts both the URL sent to
browsers and the IP the server socket listens to.  There are caveats
about valid names, valid IPs, and HSTS preload in the README.
HSTS-type errors in both Chrome and Firefox are detected automatically
and translated into friendlier errors with a short link to the
documentation.

Closes jasmine#42
Replaces PR jasmine#43
One part of solving shaka-project/shaka-player#5547
@joeyparrish
Copy link
Member Author

jasmine-browser-runner also seems to only run a single browser. That will also need to be addressed.

joeyparrish added a commit to joeyparrish/jasmine-browser-runner that referenced this issue Mar 8, 2024
This adds both TLS and hostname support.

TLS support is activated by specifying paths to a key and a cert.
There are caveats about self-signed certs in the README.

This also adds hostname support. This impacts both the URL sent to
browsers and the IP the server socket listens to.  There are caveats
about valid names, valid IPs, and HSTS preload in the README.
HSTS-type errors in both Chrome and Firefox are detected automatically
and translated into friendlier errors with a short link to the
documentation.

Closes jasmine#42
Replaces PR jasmine#43
One part of solving shaka-project/shaka-player#5547
joeyparrish added a commit to joeyparrish/jasmine-browser-runner that referenced this issue Mar 8, 2024
This adds both TLS and hostname support.

TLS support is activated by specifying paths to a key and a cert.
There are caveats about self-signed certs in the README.

This also adds hostname support. This impacts both the URL sent to
browsers and the IP the server socket listens to.  There are caveats
about valid names, valid IPs, and HSTS preload in the README.
HSTS-type errors in both Chrome and Firefox are detected automatically
and translated into friendlier errors with a short link to the
documentation.

Closes jasmine#42
Replaces PR jasmine#43
One part of solving shaka-project/shaka-player#5547
joeyparrish added a commit to joeyparrish/jasmine-browser-runner that referenced this issue Mar 8, 2024
This adds both TLS and hostname support.

TLS support is activated by specifying paths to a key and a cert.
There are caveats about self-signed certs in the README.

This also adds hostname support. This impacts both the URL sent to
browsers and the IP the server socket listens to.  There are caveats
about valid names, valid IPs, and HSTS preload in the README.
HSTS-type errors in both Chrome and Firefox are detected automatically
and translated into friendlier errors with a short link to the
documentation.

Closes jasmine#42
Replaces PR jasmine#43
One part of solving shaka-project/shaka-player#5547
@joeyparrish
Copy link
Member Author

The maintainers of jasmine-browser-runner are resistant to adding support for multiple browsers running in parallel, even if I provide a PR. (And I had this working locally.) They just don't seem to want the complexity that they feel would come with that functionality. So we may have to go another direction.

I'm currently investigating Web Test Runner, which is built to launch real browsers, just like Karma was. It supports both Selenium and WebDriverIO as WebDriver clients to launch browsers.

Needs met by Web Test Runner:

  • Real browser testing through Selenium grid so we can test video playback on real devices
  • Testing in-browser through a URL, instead of through WebDriver commands that manipulate the browser
  • Serve tests over HTTPS with a custom certificate * [ ] A way for the in-browser tests to invoke the WebDriver screenshot command in the test runner and have the runner compare that screenshot to the reference screenshots (today we have a custom Karma middleware that the tests can call into)

Needs, not sure how to get from Web Test Runner:

  • Machine-readable test coverage reports
  • Web server plugin to trigger WebDriver screenshots from test code

Wants:

  • Support for our current test framework, Jasmine (not critical)
  • Test coverage output in junit format, for compatibility with current tooling (not critical)
  • A way to run local browser tests over WebDriver, for local screenshot tests (strong preference)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: P3 Useful but not urgent type: CI An issue with our continuous integration tests
Projects
None yet
Development

No branches or pull requests

1 participant