test: [M3-7667] - (Proof of Concept) Add tagging capability for Cypress tests

[jdamore-linode]

Description 📝

Proof of concept to give us the capability to apply arbitrary tags to our Cypress tests and run Cypress against subsets of tags.

Motivation

The main goal of this effort is to give us more flexibility in how/when we run our tests. M3-7667 and M3-8072 explore some of the challenges we're facing as our test suite grows and as more teams rely on Cloud and its tests for their own purposes.

Usage

Tagging Tests

When developing tests, we can tag them using cy.tag or cy.addTag. Tags can be applied to individual tests, or to multiple tests using beforeEach hooks. cy.tag will replace any tags that have already been set, while cy.addTag will add tags in addition to any that may have been set via hooks.

A quick example of what this looks like:

describe('a group of tests', () => {
  beforeEach(() => {
    cy.tag('feat:linodes', 'method:e2e');
  });
  
  it('tests a thing', () => {
    // This test will have 'feat:linodes' and 'method:e2e' tags.
  });

  it('tests a thing with mocks', () => {
    cy.tag('feat:linodes', 'method:mock');
    // This test will have 'feat:linodes' and 'method:mock' tags.
  });

  it('tests a last thing', () => {
    cy.addTag('feat:placementGroups');
    // This test will have 'feat:linodes', 'feat:placementGroups', and 'method:e2e' tags.
  });
});

Running Tagged Tests

By default, running yarn cy:run will run all tests regardless of how they're tagged. To specify which tests should run using tags, users can use the CY_TEST_TAGS environment variable:

echo "Run all the tests with 'feat:linodes' tag"
CY_TEST_TAGS='feat:linodes' yarn cy:run

Tags can be composed and negated:

echo "Run all the tests that have either 'feat:linodes' or 'feat:placementGroups' tags"
CY_TEST_TAGS='feat:linodes,feat:placementGroups' yarn cy:run

echo "Run all the tests that have 'feat:linodes' and 'feat:placementGroups' tags"
CY_TEST_TAGS='feat:linodes feat:placementGroups' yarn cy:run

echo "Run all the tests that do not have the 'feat:linodes' tag"
CY_TEST_TAGS='-feat:linodes' yarn cy:run

echo "Run all the tests that have either 'feat:linodes' or 'feat:placementGroups' tags, but not the 'method:e2e' tag"
CY_TEST_TAGS='feat:linodes,feat:placementGroups -method:e2e' yarn cy:run

Implementation

The tag utils work by modifying each test's Mocha context when Cypress executes the test, which happens before the test runs. Because cy.tag and cy.addTag are not part of Cypress's command queue, they can be called at any point in a test and will still apply.

Why Roll Our Own?

I opted to implement our own tagging mechanism rather than rely on an existing plugin or package. There are two packages that are relevant, and inspiration was drawn from each, but neither is perfectly suitable for our needs:

• cypress-tags - This works by adding a TypeScript preprocessing step that searches for patterns in the test code itself to construct the tag graph, then strips out the tagging code and passes it to Cypress for execution. I didn't like this approach because a lot of time is already required to preprocess our specs via Vite and I wanted to avoid increasing our preprocessing time even further, and because of concerns that this method could have compatibility issues with our existing TypeScript/Vite preprocessing config and setup.
• mocha-tags - This package isn't very well supported (last updated 7 years ago, no GitHub repo) and works by mutating the global it(...), describe(...), etc. functions exposed by Mocha. I did take a lot of inspiration from its tag querying/composing mechanism, however.

Downsides, Caveats, etc.

Performance

Because skipped tests are identified at runtime, Cypress must still preprocess all spec files included in its run, even if all the tests within the spec ultimately get skipped. Consequently, running the test suite with all tests skipped still takes approximately 10 minutes (when running locally on my machine) to account for the preprocessing.

Changes to cy.defer()

This implementation required changes to cy.defer() so that it consumes a function that returns a Promise rather than consuming a Promise directly. This is necessary because tests that get skipped still get executed by Cypress (without being run), so Promises that are initialized outside of Cypress's command queue get rejected when the test is skipped, resulting in errors and exceptions in certain cases (e.g. Axios requests getting aborted).

Reliance on cy.state()

This implementation requires use of the cy.state() function in order to access the tests' Mocha contexts, which is undocumented and could be changed. We're already relying on cy.state() (albeit to a much lesser extent) because it exposes the only method available to skip a Cypress test at runtime. Even if this implementation were to change, we can adapt as long as Cypress continues to expose the Mocha context.

Changes 🔄

List any change relevant to the reviewer.

• Adds tag and addTag utils, as well as global aliases cy.tag and cy.addTag
• Adds CY_TEST_TAGS environment variable to allow users to specify which tests to run using tags
• Defines a test tag type with a couple tags, but this is mostly for demonstration purposes (we should probably put more thought into what types of tags we're actually going to want)
• For testing/demonstration purposes only, applies some tags to the tests in create-linode.spec.ts

Preview 📷

Running Linode create tests that are tagged with the e2e tag:

Tag info that gets logged when Cypress starts, intended to help troubleshoot tag queries:

Error message when the user specifies an invalid tag query:

How to test 🧪

• Confirm that running yarn cy:run runs all of the tests (you can stop after a few tests have run, no need to wait for the entire suite -- CI will help us double check this, too)
• Confirm that running CY_TEST_TAGS='method:e2e' yarn cy:run -s "cypress/e2e/core/linodes/create-linode.spec.ts" runs only the tests in create-linode.spec.ts which are tagged with method:e2e
• Feel free to experiment with more advanced tag queries and other combinations of spec files

As an Author I have considered 🤔

Check all that apply

• 👀 Doing a self review
• ❔ Our contribution guidelines
• 🤏 Splitting feature into small PRs
• ➕ Adding a changeset
• 🧪 Providing/Improving test coverage
• 🔐 Removing all sensitive information from the code and PR description
• 🚩 Using a feature flag to protect the release
• 👣 Providing comprehensive reproduction steps
• 📑 Providing or updating our documentation
• 🕛 Scheduling a pair reviewing session
• 📱 Providing mobile support
• ♿ Providing accessibility support

[github-actions]

Coverage Report: ✅
Base Coverage: 82.01%
Current Coverage: 82.01%

[AzureLatte]

yarn cy:run works, Also test for create-linode.spec.ts passed
✔ All specs passed!

[jaalah-akamai]

Overall this looks good to me 👍

I believe I read that cy.state is deprecated and will be removed in future releases (albeit it seems to have stuck around for several major releases). https://docs.cypress.io/guides/references/changelog#10-5-0

cy.state('subject') is deprecated and reading from it will log a warning to the console. Prefer cy.currentSubject() instead. Addresses [#23092](https://github.com/cypress-io/cypress/issues/23092).

Unless I'm misunderstanding something?