Blocks E2E: Refactor configs and workflow

[WunderBart]

Follow-up to #46125
Closes: #46331
Related: pdV5qK-nT-p2#possible-solutions

Proposed Changes

This PR enhances the efficiency of running our Blocks E2E tests both locally and in our CI environment.

First, we have eliminated all side effects references as a continuation of #46125, ensuring they are permanently resolved. 🎉

Regarding CI improvements, we've restructured the Playwright configuration (#46331) to enable all tests to run in a single job rather than a 3D theme-based matrix. This adjustment reduces resource usage by half (10 instead of 20 shards) and accelerates test completion due to better-balanced shards (cc: @tammullen). Additionally, we introduced a Merge Artifacts step, consolidating all debug artifacts from separate shards into one zip file for convenience.

For local testing improvements, we've reduced global setup rerun times by reusing existing database snapshots created within the initial run (introduced in #46125). We've also simplified the test execution process, making it unnecessary to identify the specific project (block-theme, classic-theme, or block-theme-with-templates) the test belongs to. Running a single suite is now as straightforward as using pnpm test:e2e path-to/any-suite.spec.ts. However, executing all theme-based groups remains an option through pnpm test:e2e:block-theme/class-theme/block-theme-with-templates when required.

Testing Instructions

Ensure running tests via pnpm test:e2e, pnpm test:e2e:block-theme, pnpm test:e2e:classic-theme, pnpm test:e2e:block-theme-with-templates works as expected. Ensure all CI tests pass.

⚠️ Some tests might still be flaky. However, that flakiness is out of the scope of this PR, as it is no longer caused by the side effects. Those flaky tests need to be investigated individually.

Next Steps

• Update E2E docs.
• Explore uploading DB snapshot to the repo to reduce setup time.

[gigitux]

looking even further, do we even need the block/classic theme distinction? Do you think it's a useful thing to have?

Yeah, good point. I think that the classic/block suffixes are useful to have some separation. Otherwise, there is the risk that we will put everything in one file, and E2E tests will be huge.

[WunderBart]

looking even further, do we even need the block/classic theme distinction? Do you think it's a useful thing to have?

Yeah, good point. I think that the classic/block suffixes are useful for some separation. Otherwise, there is the risk that we will put everything in one file, and the E2E tests will be huge.

I also lean towards keeping them in separate files – it keeps the structure flat. I'll merge the block-theme-with-templates with block-theme, then!

edit:

OTOH, there are only 11 tests for the classic theme, so I propose discussing the theme-based separation in another PR, if that's OK! 😅

[gigitux]

OTOH, there are only 11 tests for the classic theme, so I propose discussing the theme-based separation in another PR, if that's OK! 😅

As I mentioned via DMs, my comment isn't a blocker 👍 On my side, this PR can be merged!

[nielslange]

@WunderBart Firstly, thank you so much for working on this! This is another great improvement to the e2e tests. Regarding using different themes, what you and @gigitux discussed above, I recently created #46746. Not directly related to the issue raised by @gigitux, but a potential speed improvement when setting up the test env.

[github-actions]

Test using WordPress Playground

The changes in this pull request can be previewed and tested using a WordPress Playground instance.
WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Test this pull request with WordPress Playground.

Note that this URL is valid for 30 days from when this comment was last updated. You can update it by closing/reopening the PR or pushing a new commit.