Add Axe/Playwright tests #165

KevinEtchells · 2024-03-28T14:41:53Z

Context

This is setting up Playwright for accessibility and end-to-end testing, using the pytest-playwright package. The accessibility tests use Axe.

The primary focus was accessibility testing, but this also serves as an MVP for additional end-to-end testing in future.

Setup

Follow the instructions in the README at django_app/tests_playwright/.

If any pages aren't working on this branch (.e.g. /documents) you may need to run poetry run python manage.py migrate.

How to test

First check that all the tests pass
Then create some accessibility issues on a page. Here's a few if you're struggling for ideas:

<h2>Main page heading, instead of a h1</h2>
<a href=""></a>
<img src="broken-and-no-alt.jpg"/>

And check the errors are reported

wpfl-dbt · 2024-05-23T05:59:49Z

django_app/tests_playwright/_signin.py

+    os.chdir('..')
+    command = ["poetry", "run", "python", "manage.py", "show_magiclink_url", email_address]
+    result = subprocess.run(command, capture_output=True, text=True)


subprocess.run takes cwd argument you can use to set the working dir. Using os.chdir also feels a little worrying to me as we already have a repo where people assume the working directory is two different places depending on whether you're front or backend. Suggest using Path(__file__) to establish this file location as a baseline and navigate from there, so like django_root = Path(__file__).parents[2] for the Django app root, I think. Can even assert (django_root / "manage.py").exists() to make sure it's working before running subprocesses.

Because the working dir for this is tests_playwright it felt safest to navigate one step back from there, rather than outwards in (if that makes sense) - i.e. tests_playwright will always be an immediate child dir of django_app unless we explicitly move these tests. And so it should work regardless of what the overall working dir is. Unless I've missed something?

os.chdir() will work relative to the current working directory, right? Run pytest --collect-only from root -- your playwright tests will come up in the list of tests to run. I'm not certain, but if I ran them from there the working directory would be root, not the directory of this test, right?

In short, I think you can't guarantee the user's working directory when they run this, but you can guarantee where the file is saved, so use that to define the working directory for subprocess.run.

Ah, with you, thanks @wpfl-dbt. That's now updated!

wpfl-dbt · 2024-05-23T06:09:41Z

django_app/tests_playwright/README.md

+
+## Setup
+
+`poetry run playwright install`


It's hard to know if it's just my local being a bit out of date but as I ran poetry install before running this command I got told the lockfile within the Django app was out of date. Might want to check at your end with poetry lock --no-update.

I've now updated the poetry.lock file

wpfl-dbt · 2024-05-23T06:26:14Z

django_app/tests_playwright/README.md

+
+The tests are currently located at `django_app/tests_playwright`. All commands below should be run from this directory.
+
+## Setup


This all assumed the django app is running locally... on 8090. This isn't in the app documentation, which will bring it up on 8000. Suggest adding...

docker-compose up -d db minio poetry run python manage.py runserver 8090

...to your instructions.

Good point @wpfl-dbt.

I've now added a _settings.py file with BASE_URL and added this as a setup step in the README.

wpfl-dbt · 2024-05-23T06:27:52Z

.env.django

+
+# === Playwright Tests ===
+
+USER_EMAIL=


I haven't got tests working yet but assuming this can't be left blank, would strongly suggest giving it a sensible default, like "redbox-copilot@cabinetoffice.gov.uk", an email address already in the file.

It can't be left blank, but adding a default won't work as it needs to be whatever user you have setup. I've taken a couple of extra steps now:

Make the instruction in the README clearer

Throw an error as soon as possible if USER_EMAIL not set with a clear explanation of how to fix

wpfl-dbt · 2024-05-23T06:40:22Z

django_app/tests_playwright/_signin.py

+import os
+import subprocess
+from playwright.sync_api import Page, expect
+


This didn't work for me at all until I forcibly loaded the environment variables.

from dotenv import load_dotenv from pathlib import Path ROOT = Path(__file__).parents[1] load_dotenv(dotenv_path=ROOT / ".env", override=True)

I've added that in - thank you

wpfl-dbt · 2024-05-23T06:42:07Z

django_app/tests_playwright/test_accessibility.py

+
+axe = Axe()
+
+def test_violations(page: Page):


I didn't even need to change anything to see this works -- /documents throws a bunch of accessibility issues for me!

Ah, yes, it needed a migration. This isn't specific to this work, but I'll add a note to the main PR text in case anyone else also reviews!

wpfl-dbt

Not quite working well enough to approve, I think. I've left a bunch of comments.

KevinEtchells · 2024-05-23T07:29:34Z

Not quite working well enough to approve, I think. I've left a bunch of comments.

Thanks @wpfl-dbt, these comments are super helpful. I'm working through them now, and will reply to each one individually.

wpfl-dbt

Happy for this to go in once merge conflicts are resolved.

Might also be worth bringing in the latest linting rules from pyproject.toml and running them.

Add Poetry dependencies

KevinEtchells changed the title ~~Add Axe/Playwright tests~~ Draft: Add Axe/Playwright tests Apr 22, 2024

KevinEtchells force-pushed the feature/accessibility-testing branch from bd7a73d to e4ff98d Compare May 2, 2024 10:29

KevinEtchells temporarily deployed to release May 2, 2024 10:29 — with GitHub Actions Inactive

KevinEtchells had a problem deploying to release May 2, 2024 11:42 — with GitHub Actions Failure

KevinEtchells had a problem deploying to release May 3, 2024 08:11 — with GitHub Actions Failure

KevinEtchells force-pushed the feature/accessibility-testing branch from bf799be to 912aea2 Compare May 22, 2024 07:58

KevinEtchells had a problem deploying to release May 22, 2024 07:58 — with GitHub Actions Failure

KevinEtchells had a problem deploying to release May 22, 2024 10:51 — with GitHub Actions Failure

KevinEtchells force-pushed the feature/accessibility-testing branch from 423fbce to ec5de06 Compare May 22, 2024 11:00

KevinEtchells had a problem deploying to release May 22, 2024 11:00 — with GitHub Actions Failure

KevinEtchells force-pushed the feature/accessibility-testing branch from ec5de06 to 9fe5a59 Compare May 22, 2024 11:04

KevinEtchells had a problem deploying to release May 22, 2024 11:04 — with GitHub Actions Failure

KevinEtchells marked this pull request as ready for review May 22, 2024 11:04

KevinEtchells changed the title ~~Draft: Add Axe/Playwright tests~~ Add Axe/Playwright tests May 22, 2024

wpfl-dbt reviewed May 23, 2024

View reviewed changes

KevinEtchells had a problem deploying to release May 23, 2024 07:18 — with GitHub Actions Failure

KevinEtchells had a problem deploying to release May 23, 2024 07:26 — with GitHub Actions Failure

KevinEtchells temporarily deployed to release May 23, 2024 07:45 — with GitHub Actions Inactive

KevinEtchells temporarily deployed to release May 23, 2024 07:55 — with GitHub Actions Inactive

KevinEtchells temporarily deployed to release May 23, 2024 09:26 — with GitHub Actions Inactive

wpfl-dbt approved these changes May 23, 2024

View reviewed changes

KevinEtchells had a problem deploying to release May 23, 2024 13:51 — with GitHub Actions Failure

KevinEtchells added 8 commits May 23, 2024 14:59

Add Axe/Playwright tests

d3a5360

Add Poetry dependencies

Add support pages to accessibility tests

9693e02

Add sign-in functionality to Playwright tests

c47aabe

Create a Playwright settings file with BASE_URL

5bc99b7

Make USER_EMAIL instructions clearer

c3ee982

Update poetry.lock

0014492

Ensure environment variables are loaded in _signin.py

efe3685

Get Django dir relative to script dir

3fca7f9

KevinEtchells force-pushed the feature/accessibility-testing branch from ec24f6b to 3fca7f9 Compare May 23, 2024 14:00

KevinEtchells had a problem deploying to release May 23, 2024 14:00 — with GitHub Actions Failure

KevinEtchells merged commit 7f4b53b into main May 23, 2024
4 of 6 checks passed

KevinEtchells deleted the feature/accessibility-testing branch May 23, 2024 14:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Axe/Playwright tests #165

Add Axe/Playwright tests #165

KevinEtchells commented Mar 28, 2024 •

edited

wpfl-dbt May 23, 2024

KevinEtchells May 23, 2024

wpfl-dbt May 23, 2024

KevinEtchells May 23, 2024

wpfl-dbt May 23, 2024

KevinEtchells May 23, 2024

wpfl-dbt May 23, 2024 •

edited

KevinEtchells May 23, 2024

wpfl-dbt May 23, 2024

KevinEtchells May 23, 2024

wpfl-dbt May 23, 2024

KevinEtchells May 23, 2024

wpfl-dbt May 23, 2024

KevinEtchells May 23, 2024

wpfl-dbt left a comment

KevinEtchells commented May 23, 2024

wpfl-dbt left a comment


		The tests are currently located at `django_app/tests_playwright`. All commands below should be run from this directory.

		## Setup

Add Axe/Playwright tests #165

Add Axe/Playwright tests #165

Conversation

KevinEtchells commented Mar 28, 2024 • edited

Context

Setup

How to test

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wpfl-dbt May 23, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wpfl-dbt left a comment

Choose a reason for hiding this comment

KevinEtchells commented May 23, 2024

wpfl-dbt left a comment

Choose a reason for hiding this comment

KevinEtchells commented Mar 28, 2024 •

edited

wpfl-dbt May 23, 2024 •

edited