Comparison of Intern with other popular testing frameworks

[jason0x43]

The goal of this issue is to compare Intern to other popular testing tools to discover potential areas of improvement for Intern.

A few questions we should answer (plus anything else that seems relevant):

• What features do these tools have that Intern lacks, or only partially implements?
• What testing tasks are easier with these tools than with Intern?
• How does the interaction model of these tools (cli, interactivity, etc) compare to Intern's?

[aciccarello]

I have experience mainly with Jasmine + Karma and Jest so I'll discuss those two here. Since they are similar much of the feedback is true about both.

What features do these tools have that Intern lacks, or only partially implements?

• Focused test running (i.e. fdescribe/fit)
• Watch mode Watch mode #1020
• Jest: Fast parallel tests Enable test sharding (pooled environments) #546
• Jest: Snapshot testing
• Jest: Module mocking: seems kinda hacky to me but helpful when testing

What testing tasks are easier with these tools than with Intern?

• Working on fixing a single test or suite (combination of focused tests and watch mode)
• Configuration
  • Karma: Has a pretty standardized config between projects though I have more experience with this
  • Jest: Supports zero config running, I haven't written my own jest config but the options seemed more limited but more focused

How does the interaction model of these tools (cli, interactivity, etc) compare to Intern's?

I've used both the karma and jest CLIs but after setting up some npm scripts generally didn't use the CLI very much with the following exceptions

• Jest: grepping file names to focus tests
• Karma: overriding config for watch mode, browser to test, reporter type

Debugging in Karma is an interesting story. It is nice to be able to debug directly in the browser tools without needing to do much more than opening a tab. Jest you generally need to console.log or setup remote debugging.

[jason0x43]

Potential issues from this discovery:

Add / improve watch mode

See #1020

Implement test parallelization

The basic strategy here would be to let Intern spawn pools of worker endpoints and then feed them tests. This could use a similar implementation to the browser support mentioned in the watch mode issue. Remote sessions could be updated to wait for instructions, do tasks, and then go back to waiting.

See #803 and #546.

Implement snapshot testing

This one's a bit tricker, and probably needs its own discovery issue. Intern doesn't focus on any one UI system, so the problem scope is wide.

Module mocking

A built in mocking solution has been a long standing request. This is another issue that probably needs its own discovery issue to determine what we'd want out of a mocking solution.

Intern's own self tests now use a type-safe module mocking solution built on rewiremock. A combination of that and Sinon could serve as a default mocking toolkit. In any case, even if Intern provides a default solution, users should still be able to easily load alternatives.

Simplify working on a single test or suite

Intern currently allows working on single tests or suites via the suites and grep options. However, there have been multiple requests for some way to indicate in code that only a specific test should run (e.g., #514).

The initial effort should be to more clearly document, at least in Intern's "how to" page, how to restrict test execution using suites and grep.

The more involved effort should be to implement a solution for restricting test execution. This solution would work by looking for "only" makers, either during suite/test registration or just before tests are run. If any such makers were seen, only the marked tests/suites would be executed. This solution would not work on tests dynamically generated during a test run, but it would work for tests generated by plugins in suiteAdd event handlers.

Configuration improvements

Add conventional behavior to Intern to allow it to run without configuration. The main requirement would be some default suite loading behavior. For example, Intern could have default suites values:

"suites": [
	"src/**/*.spec.js",
	"tests/unit/**/*.js"
],
"functionalSuites": [
	"tests/functional/**/*.js"
]

Intern's suite loading logic could be updated to automatically try ".ts" files if ".js" files weren't present.

Interactive debugging

A low bar task for this would be to document how to use Intern with Node's --inspect functionality.

[rorticus]

Thoughts from working w/ jest for the last year...

1. It's nice to not have to import the basics, like describe and it. Not having to remember the import-fu in order to start writing a test means I can create a new file and start writing, without having to go and look up the magical first few lines that get everything set up for me.

2. IDE integration - I use webstorm, and by default, it has jest integration enabled. When I open a spec file it automatically shows a little "play" button next to the suite and each individual test. Click play quickly and easily runs the test. I can even hit debug and debug a single test by inserting breakpoints.

3. This has already been stated, but mocks! Having the mocks built into jest means that I don't have to learn more. If I move from one project to the next, but they both use jest, I already know how to write tests and mock modules.

4. I'm not too concerned with configuration. While setting up jest generally seems to be less work than setting up intern, this is really only a step you need perform once, and possibly tweak later. While I think that less configuration is a good thing, I think that the previously mentioned issues are higher priority.

[jason0x43]

Those are great! Some new issues we can pull from that...

Add a default interface to the global scope

Intern should add a test interface and an assertion interface to the global scope. By default this could be bdd and expect, but could be configurable. (That will make the typings interesting...)

Include a lower-level mocking solution

Along with module mocking (replacement, really), a built-in way to quickly generate spies and stubs would be useful. Intern could expose some or all of Sinon's API for this purpose.

Create a WebStorm extension

This is related to the "VS Code extension" task in #1021. Both extensions would serve the same purpose, and would likely have very similar implementations. The extensions should allow individual tests to be executed, and the results should be displayed in the host IDE.

[aciccarello]

Intern should add a test interface and an assertion interface to the global scope. By default this could be bdd and expect, but could be configurable. (That will make the typings interesting...)

I would personally prefer to rely on auto es6 imports over using the global scope for the bdd interface. It should also make the typings simpler.

Functional Testing Comparison

One testing tool we should also talk about is Cypress. It takes a completely different approach by not using Selenium, but there might be some lessons to learn from their API.

Things I like

• Selectors are retried until timeout to help keep tests stable
• Ensures dom references don't go stale
• Explicitly defers to mocha and sinon for some of the API
• Watching and mocking HTTP requests
• Snapshot recording (both video & images) are built in

Things I don't Like

• Doesn't use promises. It makes the tests cleaner but sometimes isn't intuitive
• Some things are "too magical" and it can be hard to track down the actual functionality

[jason0x43]

I would personally prefer to rely on auto es6 imports over using the global scope for the bdd interface. It should also make the typings simpler.

Hmmm...auto imports aren't quite as user-friendly or as universally available, but they're definitely easier to implement (in that they work now -- done! 😄). It might be useful to provide an off-by-default option to enable a global interface.

Explicitly defers to mocha and sinon for some of the API

This is something we do now (explicitly deferring to chai) that's always seemed kind of kludgey to me. I think we should continue using chai, and should possibly include sinon, too, but it would be nice if we could make them feel a bit more integrated. If nothing else we could re-export them (or the relevant parts) from intern, just so we don't have to rely on a flat npm install for the packages to be available to users.

Selectors are retried until timeout to help keep tests stable

Is this different from how webdriver selectors work? (e.g., a findByWhatever call will retry the find until the element is available or a timeout occurs)

Snapshot recording (both video & images) are built in

That's definitely an area Intern do more with. Like, automatically taking screenshots for failing tests would bee pretty great.

[indolering]

It's nice to not have to import the basics, like describe and it. Not having to remember the import-fu in order to start writing a test means I can create a new file and start writing, without having to go and look up the magical first few lines that get everything set up for me.

Based of my experience with Mocha, I actually prefer not having magical test environments:

• IDE and type checkers can pickup on the exported functions/modules.
• I can just run tests in Gulp and elsewhere without having to mimic the Mocha environment.

It's easier for me to remember import {assert, suit, test } from 'intern'; than it is to lookup function definitions and mimic the test universals.

[jason0x43]

Good point. In general, less magic is better.

[aciccarello]

Is this different from how webdriver selectors work? (e.g., a findByWhatever call will retry the find until the element is available or a timeout occurs)

I think what is special about cypress is that Cypress retries the selector and assertion together. The cypress docs have an example of if you are selecting a list of items and there is a delay before it appears you can assert the number of returned items and it will still pass. Part of this is enabled because Cypress doesn't run the commands immediately.

I will say that the Cypress retryable logic is hard to remember when writing tests. For example, selecting items in a list and then selecting elements within the returned elements doesn't retry. If intern adds similar logic I would encourage keeping it simple to follow.

Cypress has a page with all the intricacies of retries in their docs.
https://docs.cypress.io/guides/core-concepts/retry-ability.html

[jason0x43]

Ah, Cypress's retry, at least for a list-of-items example, is kind of like Intern's findAllBy + pollUntil. The Webdriver find and findAll commands do retry until the basic condition (presence) is met, but Cypress's find lets you extend that condition to, say, retry until a minimum count of an item is present.

[jason0x43]

See #514 for fdescribe/fit support.

[jason0x43]

Closing this as discovered.

Outcomes:

• Support focused testing (fdescribe/fit) (Feature: suite.only() and test.only() #514)
• Support parallel testing (Enable test sharding (pooled environments) #546)
• Include a mocking facilities (Include a default mock/stub library #47, Add module mocking functionality #1140)
• Run with zero config (Run with zero config #1141)
• Support snapshot testing (Add support for snapshot testing #1142)