E2E tests are failing locally and in CI

[mokagio]

Quick summary

The E2E tests are currently failing, both locally and in CI.

I think the reason this hasn't been noticed in PRs is that the workflow step is configured with continue-on-error: true.

Steps to reproduce

Locally, run npm install && npm run package && npm run e2e.

On CI, visit a recent PR and look at its actions. For example, #56: https://github.com/Automattic/studio/actions/runs/8830011430/job/24242246592?pr=27

What you expected to happen

E2E tests should be able to create tmp sites and move forward with the testing.

What actually happened

Locally, you can see the app being stuck on the "Add your first site" screen:

I assume the same happens in CI, based on the way the tests fail, but I have no way to verify it.

Impact

One

Available workarounds?

There is no user impact

Platform

No response

Logs or notes

Here's my local test log

➜ npm run e2e


> studio@1.0.0 e2e
> npx playwright install && npx playwright test


Running 4 tests using 1 worker

  ✓  1 app.test.ts:18:6 › Electron app › should ensure app title is correct. (242ms)
  ✘  2 sites.test.ts:34:6 › Servers › create a new site (15.6s)
  ✘  3 sites.test.ts:60:6 › Servers › edit site's settings in wp-admin (505ms)
  ✘  4 sites.test.ts:77:6 › Servers › delete site (5.5s)


  1) sites.test.ts:34:6 › Servers › create a new site ──────────────────────────────────────────────

    Error: locator.inputValue: Error: Node is not an <input>, <textarea> or <select> element
    Call log:
      - waiting for getByRole('dialog', { name: 'Add a site' }).getByLabel('Local path')


      38 | 		await modal.siteNameInput.fill( siteName );
      39 | 		await modal.clickLocalPathButtonAndSelectFromEnv();
    > 40 | 		expect( await modal.localPathInput.inputValue() ).toBe( tmpSiteDir );
         | 		                                   ^
      41 | 		await modal.addSiteButton.click();
      42 |
      43 | 		const sidebarButton = sidebar.getSiteNavButton( siteName );

        at /Users/gio/Developer/a8c/studio/e2e/sites.test.ts:40:38

  2) sites.test.ts:60:6 › Servers › edit site's settings in wp-admin ───────────────────────────────

    Error: page.goto: net::ERR_CONNECTION_REFUSED at http://localhost:8881/wp-admin/options-general.php
    Call log:
      - navigating to "http://localhost:8881/wp-admin/options-general.php", waiting until "load"


      66 |
      67 | 		// page.goto opens a browser
    > 68 | 		await page.goto( wpAdminUrl + '/options-general.php' );
         | 		           ^
      69 | 		const siteTitleInput = page.getByLabel( 'Site Title' );
      70 | 		await siteTitleInput.fill( 'testing site title' );
      71 | 		await siteTitleInput.press( 'Enter' );

        at /Users/gio/Developer/a8c/studio/e2e/sites.test.ts:68:14

  3) sites.test.ts:77:6 › Servers › delete site ────────────────────────────────────────────────────

    Error: Timed out 5000ms waiting for expect(locator).toBeVisible()

    Locator: getByRole('dialog', { name: 'Delete test-site-1ad71ca0-c394-4059-ae58-59195465f3af' })
    Expected: visible
    Received: hidden
    Call log:
      - expect.toBeVisible with timeout 5000ms
      - waiting for getByRole('dialog', { name: 'Delete test-site-1ad71ca0-c394-4059-ae58-59195465f3af' })


       at page-objects/settings-tab.ts:43

      41 | 		await this.deleteButton.click();
      42 | 		const modal = new DeleteSiteModal( this.page, this.siteName );
    > 43 | 		await expect( modal.locator ).toBeVisible();
         | 		                              ^
      44 | 		return modal;
      45 | 	}
      46 | }

        at SettingsTab.openDeleteSiteModal (/Users/gio/Developer/a8c/studio/e2e/page-objects/settings-tab.ts:43:33)
        at /Users/gio/Developer/a8c/studio/e2e/sites.test.ts:81:17

  Slow test file: sites.test.ts (21.6s)
  Consider splitting slow test files to speed up parallel execution
  3 failed
    sites.test.ts:34:6 › Servers › create a new site ───────────────────────────────────────────────
    sites.test.ts:60:6 › Servers › edit site's settings in wp-admin ────────────────────────────────
    sites.test.ts:77:6 › Servers › delete site ─────────────────────────────────────────────────────
  1 passed (29.7s)

[mokagio]

I updated #38 to upload the Playwright traces.

They can be downloaded individually and inspected with

npx playwright show-trace ~/Downloads/trace.zip

Here's what the one "create a new site" shows:

Two things to notice:

• The console shows that the element exists, but it's detached from the DOM. I'm not sure what this means practically, given that the code should return the Onboarding component if no site is present, and that's what one experiences locally as per my screenshot above.
• There doesn't seem to be any view in the trace. I don't know why that's the case, but that's also what happens when running `npm run e2e -- --ui

[mokagio]

I have a crude fix for the onboarding issue at mokagio/e2e-fix-attempt. The branch is a WIP mess of experiments and comments, so I don't recommend using it. Just leaving a breadcrumb in case I get sidetracked from other tasks.