Option to abort on first failure

[KittyGiraudel]

Current behavior:

All tests are run even if one fails.

Expected behavior:

Providing an option to abort the test suite as soon as a test fails.

How to reproduce the current behavior:

Make a test fail.

Environment

• Cypress Version: 0.19.2

[01taylop]

The use-case I have for this is that I need my authentication tests to pass first - if they don't all subsequent tests will fail. Although the other tests will fail in the before hooks, it still slows things down unnecessarily (30 seconds per test file).

I have to name my test 1auth so that it runs first and whitelist a cookie which contains a token required for the other tests to pass.

Cypress.Cookies.defaults({
  whitelist: 'token',
})

I was hoping to do something like this in my auth tests to stop running tests on fail;

  before(() => {
    Cypress.on('fail', () => {
      Cypress.runner.stop()
    })
  })

It does stop the test runner, but it doesn't fail any tests so I get 3/3 tests passed and a green build which is not the intended outcome. I also tried adding am intentional failing test before stopping cypress, but then the test runner continues.

[danh91]

I have a temporary solution that I think can help someone.

I just add an after each in my utilities file imported by all my specs.

afterEach(function() {
  if (this.currentTest.state === 'failed') {
    Cypress.runner.stop()
  }
});

I take advantage of Mocha offering a way to check the state of the last test.
And the nice part is that the process exit with a failing code (not 0)

[sanderdejong88]

@danh91 I've been following this thread for months, and finally someone found a way! Awesome!
I've put your code in the support file which also seems to work.

[harvitronix]

This no longer works for me with Cypress 3.x.

[jennifer-shehane]

@harvitronix We changed how specs are run in 3.0 so that the files are run isolated. This means the before runs before each spec file - so I am guessing that it is stopping and failing that 1 spec file and then continuing to the next file. Is this what is happening?

[brian-mann]

Yes we've waited to implement this feature due to how aborting is much more complex now.

We likely need to allow for multiple abort strategies

abortStrategy: 'spec' | 'run'

One would abort just the spec, one would abort the entire run.

This becomes even more complex with parallelization.

[harvitronix]

@jennifer-shehane that's exactly right. Totally understand it gets much more complex now. Would love to have the options you describe, @brian-mann.

[mohsenny]

I tried your solutions and the runner correctly stops.
However, I'm looking for a way to only stop the current test scenario/test file/describe(), but let the runner keep running to execute the rest of scenarios/test files/describe()s.
Any suggestion?

An example:

As you can see, the 3rd test file (Device Type) failed and the runner correctly didn't execute the rest of the test cases in that file, but it also didn't go any further with the 4th test file (Device) either.

[drumheadrob]

Hi, loving Cypress, but not being able to stop our CI build for a failing test is stopping me from being able to use this in CI, and will especially be a problem in CD.

I've searched quite thoroughly and hope I'm missing something somewhere, but it seems like this is related to this thread?

If so, is there any progress / planned work on this solution? Cheers.

[bahmutov]

@drumheadrob your issue is very different - you are saying that the cypress run does NOT exit with code 1 after all tests complete and some fail?!

[jennifer-shehane]

We are currently working on the underlying processes needed to 'cancel' a run and any other queued spec files, which is a precursor to the work needed to 'abort' the rest of the tests on first failure.

[pacop]

Some news right here? I am facing the very same problem, we were not able to use cypress on our CI.

[nantoniazzi]

Are you still working on this feature ?
The more we add tests with Cypress, the more it becomes unusable in our CI

[cmcnicholas]

We have the same situation, paying for cypress dashboard on private repos but our tests (now ~400 or so) are taking more and more time to complete. Much needed to stop early so our pipeline is freed up for the next PR to execute.

[vladarefiev]

It will be cool feature

[Josh68]

Another hack for making the spec fail (as I'd expect):

if (
  this?.currentTest?.state === "failed" // plus more conditions for retries
) {
...
  if ("state" in cy && typeof cy.state === "function") {
    cy.state("test").state = "failed";
  }
...
}

[csvan]

Just a note: the above workaround implies you are not using arrow functions for your test definitions.

[Josh68]

Just a note: the above workaround implies you are not using arrow functions for your test definitions.

Correct, it should be clear to anyone following along with this that for Mocha.Context to be available in the callback, you can't use an arrow function. Also, if you want to write a function you can pass by name as a callback, type it like so:

const failEarly: Mocha.Func = function failEarly() { // could also be anonymous
  // `this` is Mocha.Context
  ...
}

[Josh68]

Another hack for making the spec fail (as I'd expect):

if (
  this?.currentTest?.state === "failed" // plus more conditions for retries
) {
...
  if ("state" in cy && typeof cy.state === "function") {
    cy.state("test").state = "failed";
  }
...
}

I'm now finding this really doesn't do anything, AFAICT, and it looks suspiciously wrong anyway. I'm going to stop jumping through hoops assuming there are things I can do that aren't in the public API.

[Josh68]

@jennifer-shehane, can you comment on whether any of these strategies ought to work with v12+? I've been upgrading from v9, and at this point, don't have a strategy that aborts early on a failed test, marks the spec as failed, marks remaining tests as either Pending or Skipped, and then moves on (some may want the entire suite to stop, but we weren't doing that in the past).

Thank you

[Josh68]

Based on this example by @drumslave-git, I came up with something that I think is working globally:

export const failEarly = (runModeOnly = true) => {
  if (runModeOnly && Cypress.config("isInteractive")) return; // headless runs only
  const failedTest: Record<string, string | undefined> = {};
  beforeEach(function() {
    const specName = Cypress.spec.name;
    if (failedTest[specName]) {
      cy.log(
        `*** Spec has failed. You will see an error emitted from beforeEach, but only need to fix failing tests ***`
      );
    }
    cy.wrap(failedTest[specName]).should("be.undefined");
  });
  afterEach(function() {
    const retryCount = Cypress.currentRetry;
    const specName = Cypress.spec.name;
    if (
      this?.currentTest?.state === "failed" &&
      Number.isInteger(retryCount) &&
      getRunTypeRetries() <= retryCount
    ) {
      failedTest[
        specName
      ] = `Previous test, "${this.currentTest.title}", failed. This spec will be aborted.`;
    }
  });
};

getRunTypeRetries looks like this

const getRunTypeRetries = () => {
  const retryConfig = Cypress.config("retries");
  let configuredRetries = 0;
  if (typeof retryConfig === "number" && Number.isInteger(retryConfig)) {
    configuredRetries = retryConfig;
  }
  if (typeof retryConfig === "object" && null !== retryConfig) {
    if (
      retryConfig.runMode &&
      Number.isInteger(retryConfig.runMode) &&
      !Cypress.config("isInteractive")
    ) {
      configuredRetries = retryConfig.runMode;
    }
    if (
      retryConfig.openMode &&
      Number.isInteger(retryConfig.openMode) &&
      Cypress.config("isInteractive")
    ) {
      configuredRetries = retryConfig.openMode;
    }
  }
  return configuredRetries;
};

I imported failEarly into my e2e.ts and called it with

const runModeOnly = !Cypress.env("BROWSER_FAIL_EARLY") == true;
failEarly(runModeOnly);

with the idea that I can add CYPRESS_BROWSER_FAIL_EARLY=true before a cypress:open, if desired.

Note, I'm using cypress-terminal-report to display those cy.logs on headless runs

[w5l]

Options with Cypress.runner.end() or Cypress.runner.stop() don't work for me as they stop cypress before uploading test results. After some puzzling I came up with the following. This requires no extra setup, magic variables etc, just a simple afterEach call.

If the current test is failed, it cancels all subsequent tests within the same container. Since I only use one level of nesting (ie describe( it(), it(), it(), ...)), this is enough. All tests (within the spec) after a failed are now skipped.

This uses the Mocha bail option.

Enables or disables bailing on the first failure.

Works either on the global Cypress e2e.ts (or .js) level, or in individual specs:

before(() => {
  // Skip subsequent tests in spec when one fails.
  (cy.state('runnable').ctx as Mocha.Context).currentTest.parent.bail(true);
});

---

Update: Note that this is a fix for an older Cypress version version, recent comments suggest it no longer works, see #518 (comment)

Update 2: I was told that cy.runnableCtx() can be used instead of the deprecated cy.state(), resulting in the command cy.runnableCtx().currentTest.parent.bail(true);. That function is defined here:

cypress/packages/driver/src/cypress/cy.ts

Lines 1075 to 1079 in 22aacb1

	private runnableCtx (name) {
	ensureRunnable(this, name)
	return this.state('runnable').ctx
	}

Note that it is marked private so it can be changed at any time. Haven't tested myself, ymmv. Can't be much worse than using DOM selectors to find and click a UI button to stop, really 😉

[scmrus]

Is that the solution for 12.6.0 and later: https://docs.cypress.io/guides/guides/command-line#auto-cancel-after-runs ?

[Josh68]

Is that the solution for 12.6.0 and later: https://docs.cypress.io/guides/guides/command-line#auto-cancel-after-runs ?

Thanks for pointing this out. Will give it a try when we have a chance to upgrade

[BloodyRain2k]

I've spent half the day trying to find a way to prevent Cypress from continuing after failure, simply to preserve the debug data that what I'm testing left me for debugging.

After what feels like a dozen different approaches that didn't work at all or not for my case, I went and bruteforced it this way.

    if (Cypress.env("pause_after_fail")) {
      Cypress.on("test:after:run", (attrs, test) => {
        if (test.isFailed()) {
          const stop = window.top?.document.querySelector(".reporter button.stop") as HTMLElement;
          stop.click();
        }
      });
    }

What I did NOT expect was this to work.

"Best case" expectation was that it does click the "Stop" button, but keeps "rolling" from momentum (it's dozen Promise-chains).
But it doesn't. It just stops instantly and I can inspect my debug output without any worry.

If you're not using TypeScript, you can even reduce it to just .querySelector()?.click().
That storing as HTMLElement was simply so TypeScript would shut up about .click() not existing.

[machineghost]

before(() => {
  // Skip subsequent tests in spec when one fails.
  (cy.state('runnable').ctx as Mocha.Context).currentTest.parent.bail(true);
});

It seems cy.state no longer exists, so this solution is no longer viable.

Is that the solution for 12.6.0 and later: https://docs.cypress.io/guides/guides/command-line#auto-cancel-after-runs ?

It seems that auto-cancel-after-runs only works with the Cypress Cloud, and if you try to just cypress run --auto-cancel-after-failures 1 you get:

You passed the --ci-build-id, --group, --tag, --parallel, or --auto-cancel-after-failures flag without also passing the --record flag.

The --auto-cancel-after-failures flag you passed was: 1

These flags can only be used when recording to Cypress Cloud.

But the good news is that BloodyRain2k's solution did work. You can either gate it with an environment variable (as in the original code), or make a function out of it:

export default function enableFastFail() {
  Cypress.on('test:after:run', (attrs, test) => {
    if (test.isFailed()) {
      const stop = window.top?.document.querySelector(
        '.reporter button.stop'
      ) as HTMLElement;
      stop.click();
    }
  });
};

That way, when you want the runner to stop after the first fail, you just temporarily import enableFastFail and run it in the root of your test file. Thanks BloodyRain2k!

[NigelCunningham]

This works for me with Cypress 13.3.2

context('Context Name', () => {
  describe('Tests', () => {
    afterEach(() => {
      if (cy.runnableCtx().currentTest.isFailed()) {
        const stop = window.top?.document.querySelector(".reporter button.stop");
          stop.click();
      }
    })
  })
})

[vitaliy4us]

As far as I can see the most of solutions are about Cypress Runner:
const stop = window.top?.document.querySelector(".reporter button.stop")
But in this case it is almost useless. Headless mode is the point where we really need it, as it is used in CI tests runs and can save a lot of execution time. Especially for long e2e tests

[bahmutov]

Why, it should work in headless mode tooSent from my iPhoneOn Jan 20, 2024, at 09:37, vitaliy4us ***@***.***> wrote: As far as I can see the most of solutions is about Cypress Runner: const stop = window.top?.document.querySelector(".reporter button.stop") But in this case it is almost useless. Headless mode is the point where we really need it, as it is used in CI tests runs and can save a lot of execution time. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[vitaliy4us]

@bahmutov is this button accessible in headless mode? My assumption was that Cypress Test Runner UI is for headed mode only.

[NigelCunningham]

Yes, it worked for me.

[mihhail-m]

This is what I imagine was/is a business plan of the Cypress:

Step 1: Group existing smaller open source testing libraries into one big bloated package
Step 2: Expose limited functionality in the free version
Step 3: Hire marketing/sales people to sell your bs to clueless managers
Step 4: Ask people for money if they want features that are provided by the smaller open source libraries included into Cypress to be expose to them
Step 5: Blueball the rest of the community
Step 6: profit

This is 2024 and there is still no simple way to simply stop a test 😂 What a waste of money and effort of the open source community. I believe there is more marketing to this product than actual engineering. Absolute disgrace of the open source. If you reading this than consider switching to another testing framework.

[csvan]

@mihhail-m not to simp for the cypress team but they need to make money too, just like you and me. We are happy that so much of Cypress is working freely, and since it is open source nothing stops you from sending a PR that does what you want.

[machineghost]

nothing stops you from sending a PR that does what you want.

It seems the Cypress team doesn't want this feature. If someone doesn't want a feature, you can submit a million PRs, but it won't do anyone any good.

Personally I think that if people don't like the Cypress business model, they should use a different library. It's very possible the grass is greener, and I can confirm that there's at least one viable Cypress competitor out there with support for failing fast "out of the box".

Of course, more competition in the space is always good, so I hope someday the Cypress team will prioritize user experience, and address this ticket.

[philly-vanilly]

lol what an abysmal management. a simple abort function as part of an (advanced!) business plan starting at 3000$ annually? when I noticed a while ago that Gleb (who was always helping about cypress with passion) is not on your team anymore, I thought something is fishy. I did some research and look, in addition to this, cypress has blocked sorry-cypress from current versions overriding existing docker images. no prior notice was given. it seems you guys are in financial trouble and are getting pressured to produce some revenue. but instead of being competitive you choose the Oracle business model of locking existing users in. well let me tell you something, Cypress is not a database and a migration is just a matter of 1-3 sprints for most projects. clearly, the 3000$ are better invested in replacing the lib than being stuck with a parasitic company (a term you have used for sorry-cypress) like yours. if you believe the rumors cypress as an independent company will probably not exist soon (my bet is you will be probably be incorporated into some other cloud service). now if you were a software architect and had to make a decision: to integrate cypress enterprise which might become obsolete in the foreseeable future, or to migrate to Playwright, what would you do?

[philly-vanilly]

    if (test.isFailed()) {
      const stop = window.top?.document.querySelector(
        '.reporter button.stop'
      ) as HTMLElement;
      stop.click();
    }

I had a problem with this solution as it prevented tests from being retried. If anyone else also suffers from this, you can filter error messages on the test as a temporary solution (long term solution would be ditching Cypress):

afterEach(() => {
  const currentTest = ((cy as any).runnableCtx() as Mocha.Context).currentTest;

  // let login tests retry, do not abort
  if (currentTest.isFailed() && !currentTest.err?.message.includes('[data-cy="user-menu"]')) {
    const stop = window.top?.document.querySelector('.reporter button.stop');
    stop.click();
  }
});

[seyfer]

@mihhail-m I bet you meant playwright as an alternative

[csvan]

if you believe the rumors

That was posted over a year ago, has clearly not aged well, and has tons of comments pointing out how unfounded, full of untruths, and generally out of touch it is. Cypress is still here and by all accounts doing quite well.

Don't spread FUD, do better.

[philly-vanilly]

if you believe the rumors

That was posted over a year ago, has clearly not aged well, and has tons of comments pointing out how unfounded, full of untruths, and generally out of touch it is. Cypress is still here and by all accounts doing quite well.

Don't spread FUD, do better.

yeah I also noticed the influx of paid cypress ambassadors, perhaps they can found a band "cypress shill" or something.

of course cypress is going to exist, there are way too many projects that switched from selenium or protractor to cypress years ago, so cypress can milk them for a while. but you don't need to be a fortune teller to see that the quality of the product is going downhill, especially taking the alternative (I will not say competition since Microsoft is not even earning much from it) in mind. Meanwhile cypress made a pact with venture capital and is pressured to squeeze profit with no regard to customer experience (regardless of being a free or paid customer). Don't get me wrong, I don't like tech megacorps and I wish tech startups that invest in open source could thrive easier (and their devs get paid their fair salaries). But my impression in this case is that most money is now spent on management etc and not development and that is a red flag for me.

[philly-vanilly]

If someone doesn't want a feature, you can submit a million PRs, but it won't do anyone any good.

@machineghost

Actually, since Cypress is open source with MIT license, thus you are allowed to clone and manipulate the code, I would welcome if someone made changes to Cypress in form of a https://www.npmjs.com/package/patch-package file and posted it in this thread. pnpm users don't even need patch-package. Surely a feature that simple should not require too many files changed.

[csvan]

yeah I also noticed the influx of paid cypress ambassadors, perhaps they can found a band "cypress shill" or something.

Cypress ambassadors are not paid. Can you at least try to apply some critical thinking before just dumping unsourced Reddit rants and "rumours" which do nothing but mislead people?

You can read about the ambassador programme here - notice the complete absence of any monetary compensation. Note also that Cypress is far from the only company to have something similar.

https://www.cypress.io/ambassadors

But my impression in this case is that most money is now spent on management etc and not development and that is a red flag for me.

You sure do like basing things on feelings, "impressions" and unfounded personal opinions rather than available facts. Again, this only leads to people being misled and does not help anyone. FUD is toxic.

I would welcome if someone made changes to Cypress

Any particular reason you are not willing to contribute it yourself, since you are clearly quite passionate about it?

[EugeneSEP]

I'm not sure if this is the right place to ask, but I'm hoping someone here might have had a similar experience and could offer some advice. I have cypress, allure reporter and cypress-fail-fast
plugin
I'm encountering an issue with the Allure reporter in Cypress. I've set it up along with the fast-fail plugin, but my Allure report is only displaying failed and passed tests, excluding others.

How can I ensure that all tests, regardless of their outcome, are included in the report? Any guidance would be appreciated!