Refactor Accessibility Testing #3814

jperson1 · 2024-05-09T14:47:46Z

Refactor Accessibility Testing

Issue: #3618

Drafting initially to ensure GHA workflows work properly.

Brief overview:

Added cypress-axe to run accessibility tests
Added an accessibility cypress spec to check all the things! The home/submission/access/search pages are all covered.
Changed the load_fixtures command to add a few more SACs per user, and some dummy dissemination data.
Changed the PR workflow to include the new tests, so they run every time we make a PR/merge a branch.

Changes:

Package/package lock changes for cypress-axe.
New accessibility cypress spec.
Cypress log functions to properly display accessibility errors in table format (includes links to the requirements).
Update to load_fixtures management command to support the new tests.
Allow access to SACs without permission if DISABLE_AUTH=True. Similarly, allow access to the homepage without redirection if DISABLE_AUTH=True.
Minor full submission changes to line up with new HTML id's.
All the HTML/SCSS changes to get accessibility up to date. See full log below.

How to test:

Please follow the instructions in the "Accessibility" section of testing.md. But basically:

Run with DISABLE_AUTH=True
Run the load_fixtures management command in the docker container after "logging in" once.
Run the accessibility spec via Cypress.

All the accessibility changes by page:

Sometimes, you just need h1's.

All pages:
1. Buttons in pop out primary nav should have aria labels or discernible text.
4. Removed all `aria-controls` references to 'report-cancel'. It existed briefly, then didn't. 

Home page:
1. <ul> elements in homepage should only contain <li>.
2. Remove the `aria-controls` for the login modal if auth is disabled. `aria-controls` should always point to something.

Search pages:
1. The orientation modal link should have an aria label.
2. The h2 header should be an h1.
5. The grey "search" text should be darker.
6. The search info boxes should have higher level headers.

Summary page:
1. The title needs to be an h1.
2. The greyed out text for empty findings/notes/whatever should be a little darker.

Submission checklist page:
1. Key info boxes should have unique aria labels
2. The top level <ul> should contain only <li>'s. Divs are swapped over.

Audit info page:
1. The dollar threshold doesn't have an implicit label, so it's been given an explicit `aria-label`.

Report upload page:
1. Process list h4's should be h2's (process order).
2. Page number inputs were missing implicit labels. Ids changed to match the corresponding label 'for' element.
3. Moved the <ol> down a few levels to ensure it only directly contains <li>'s.

Workbook upload page:
1. <h1> needs to exist
2. To keep ordering, <h4> becomes <h2>

Access management pages:
1. <h1> needs to exist
2. Remove `aria-controls` that don't point at anything.

Lock for certification page:
1. <h1> needs to exist
2. Remove `aria-controls` that don't point at anything.

Certification pages:
1. <h1>'s need to exist.
2. <h3>'s need to be <h2>'s. Class changes to maintain font size.

Some funky fresh screenshots:

Cypress accessibility test results (headless mode):

An accessibility error during Cypress tests (standalone mode):

PR checklist: submitters

PR checklist: reviewers

Pull the branch to your local environment and run make docker-clean; make docker-first-run && docker compose up; then run docker compose exec web /bin/bash -c "python manage.py test"
Manually test out the changes locally, or check this box to verify that it wasn’t applicable in this case.
Check that the PR has appropriate tests. Look out for changes in HTML/JS/JSON Schema logic that may need to be captured in Python tests even though the logic isn’t in Python.
Verify that no Git surgery is necessary at any point (such as during a merge party), or, if it was, repeat the testing after it’s finished.

The larger the PR, the stricter we should be about these points.

github-actions · 2024-05-09T14:48:33Z

Terraform plan for meta

No changes. Your infrastructure matches the configuration.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Warning: Argument is deprecated

  with module.s3-backups.cloudfoundry_service_instance.bucket,
  on /tmp/terraform-data-dir/modules/s3-backups/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

✅ Plan applied in Deploy to Development and Management Environment #679

github-actions · 2024-05-09T14:48:37Z

Terraform plan for dev

No changes. Your infrastructure matches the configuration.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Warning: Argument is deprecated

  with module.dev.module.cg-logshipper.module.s3-logshipper-storage.cloudfoundry_service_instance.bucket,
  on /tmp/terraform-data-dir/modules/dev.cg-logshipper.s3-logshipper-storage/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

(and 4 more similar warnings elsewhere)

✅ Plan applied in Deploy to Development and Management Environment #679

github-actions · 2024-05-09T15:08:50Z

File	Coverage		Missing
All files	`87%`	✅
api/serializers.py	`88%`	✅	`177-178` `183` `188`
api/test_views.py	`95%`	✅	`103`
api/uei.py	`88%`	✅	`87` `118-119` `163` `167-168`
api/views.py	`98%`	✅	`198-199` `337-338`
audit/forms.py	`60%`	❌	`31-38` `109-116`
audit/intake_to_dissemination.py	`88%`	✅	`57-62` `264` `308-316`
audit/mixins.py	`96%`	✅	`28`
audit/test_commands.py	`91%`	✅
audit/test_intakelib.py	`88%`	✅	`154-158`
audit/test_manage_submission_access_view.py	`98%`	✅	`15` `19`
audit/test_mixins.py	`90%`	✅	`159-160` `164-166` `254-255` `259-261`
audit/test_validators.py	`95%`	✅	`439` `443` `611-612` `851` `858` `865` `872` `1117-1118` `1149-1150` `1175-1180`
audit/test_views.py	`98%`	✅	`132`
audit/test_workbooks_should_fail.py	`88%`	✅	`58` `87-88` `92`
audit/test_workbooks_should_pass.py	`87%`	✅	`59` `74-76`
audit/utils.py	`86%`	✅	`9` `19` `60-62` `65`
audit/validators.py	`93%`	✅	`138` `190` `272` `412-413` `428-429` `512-513` `615-619` `624-628` `644-653`
audit/cross_validation/additional_ueis.py	`93%`	✅	`33`
audit/cross_validation/check_award_ref_declaration.py	`90%`	✅
audit/cross_validation/check_award_reference_uniqueness.py	`93%`	✅
audit/cross_validation/check_certifying_contacts.py	`87%`	✅
audit/cross_validation/check_findings_count_consistency.py	`91%`	✅
audit/cross_validation/check_ref_number_in_cap.py	`90%`	✅
audit/cross_validation/check_ref_number_in_findings_text.py	`90%`	✅
audit/cross_validation/errors.py	`78%`	❌	`30` `77`
audit/cross_validation/naming.py	`93%`	✅	`229`
audit/cross_validation/submission_progress_check.py	`91%`	✅	`83` `126` `174` `182-183`
audit/cross_validation/tribal_data_sharing_consent.py	`81%`	❌	`33` `36` `40`
audit/cross_validation/validate_general_information.py	`65%`	❌	`77` `81-84` `96` `99`
audit/fixtures/dissemination.py	`71%`	❌	`38`
audit/fixtures/single_audit_checklist.py	`55%`	❌	`160-197` `245-254`
audit/intakelib/exceptions.py	`71%`	❌	`7-9` `12`
audit/intakelib/intermediate_representation.py	`91%`	✅	`27-28` `73` `91` `129` `200-203` `212-213` `283-284`
audit/intakelib/mapping_audit_findings.py	`97%`	✅	`55`
audit/intakelib/mapping_audit_findings_text.py	`97%`	✅	`54`
audit/intakelib/mapping_federal_awards.py	`93%`	✅	`92`
audit/intakelib/mapping_util.py	`79%`	❌	`21` `25` `29` `63` `99` `104-105` `114-120` `130` `145` `150`
audit/intakelib/checks/check_all_unique_award_numbers.py	`79%`	❌	`24`
audit/intakelib/checks/check_cluster_names.py	`75%`	❌	`20-25`
audit/intakelib/checks/check_cluster_total.py	`95%`	✅	`99`
audit/intakelib/checks/check_finding_reference_pattern.py	`73%`	❌	`31` `41-42`
audit/intakelib/checks/check_has_all_the_named_ranges.py	`95%`	✅	`66`
audit/intakelib/checks/check_is_a_workbook.py	`69%`	❌	`20`
audit/intakelib/checks/check_loan_balance_entries.py	`83%`	❌	`28`
audit/intakelib/checks/check_look_for_empty_rows.py	`91%`	✅	`18`
audit/intakelib/checks/check_no_major_program_no_type.py	`76%`	❌	`18` `27`
audit/intakelib/checks/check_no_repeat_findings.py	`88%`	✅	`21`
audit/intakelib/checks/check_other_cluster_names.py	`81%`	❌	`23` `33`
audit/intakelib/checks/check_passthrough_name_when_no_direct.py	`83%`	❌	`11` `49` `58`
audit/intakelib/checks/check_sequential_award_numbers.py	`82%`	❌	`25` `35`
audit/intakelib/checks/check_start_and_end_rows_of_all_columns_are_same.py	`89%`	✅	`14`
audit/intakelib/checks/check_state_cluster_names.py	`81%`	❌	`23` `33`
audit/intakelib/checks/check_version_number.py	`73%`	❌	`30` `40-41`
audit/intakelib/checks/runners.py	`97%`	✅	`157`
audit/intakelib/common/util.py	`90%`	✅	`22` `39`
audit/intakelib/transforms/xform_rename_additional_notes_sheet.py	`81%`	❌	`14`
audit/management/commands/load_fixtures.py	`47%`	❌	`40-46`
audit/models/models.py	`85%`	✅	`58` `60` `65` `67` `209` `215` `227` `239-242` `260` `437` `455-456` `464` `486` `584-585` `589` `597` `606` `612`
audit/views/audit_info_form_view.py	`27%`	❌	`25-74` `77-117` `120-137`
audit/views/manage_submission.py	`86%`	✅	`73-80`
audit/views/manage_submission_access.py	`98%`	✅	`113-114`
audit/views/pre_dissemination_download_view.py	`78%`	❌	`15-16` `21-22` `29-39`
audit/views/submission_progress_view.py	`89%`	✅	`117` `182-183`
audit/views/tribal_data_consent.py	`34%`	❌	`23-41` `44-79`
audit/views/unlock_after_certification.py	`57%`	❌	`28-51` `73-87`
audit/views/upload_report_view.py	`26%`	❌	`32-35` `44` `91-117` `120-170` `178-209`
audit/views/views.py	`53%`	❌	`74` `81-100` `123-124` `198-199` `220-230` `257` `268-269` `280-281` `283-287` `329-342` `345-359` `364-377` `394-400` `405-425` `452-456` `461-490` `533-537` `542-562` `589-593` `598-627` `670-674` `679-691` `694-704` `709-721` `754-768`
census_historical_migration/change_record.py	`98%`	✅	`30`
census_historical_migration/end_to_end_core.py	`33%`	❌	`56-88` `92-110` `115-146` `152-178` `227-239` `244` `254-288`
census_historical_migration/invalid_record.py	`93%`	✅	`48` `52` `56` `60` `64`
census_historical_migration/migration_result.py	`75%`	❌	`17` `21` `25` `29` `33-42` `46`
census_historical_migration/test_federal_awards_xforms.py	`99%`	✅	`214-215`
census_historical_migration/sac_general_lib/audit_information.py	`92%`	✅	`24` `78`
census_historical_migration/sac_general_lib/cognizant_oversight.py	`68%`	❌	`11`
census_historical_migration/sac_general_lib/general_information.py	`91%`	✅	`166-167` `177-178` `186-187` `195-200` `250` `319`
census_historical_migration/sac_general_lib/sac_creator.py	`90%`	✅	`34`
census_historical_migration/sac_general_lib/utils.py	`83%`	❌	`33` `60-69`
census_historical_migration/transforms/xform_remove_hyphen_and_pad_zip.py	`92%`	✅	`18`
census_historical_migration/transforms/xform_retrieve_uei.py	`67%`	❌	`10`
census_historical_migration/transforms/xform_string_to_bool.py	`87%`	✅	`17`
census_historical_migration/workbooklib/additional_eins.py	`84%`	❌	`58-60` `67-77`
census_historical_migration/workbooklib/additional_ueis.py	`77%`	❌	`27-29` `36-46`
census_historical_migration/workbooklib/corrective_action_plan.py	`83%`	❌	`47-49` `91-108`
census_historical_migration/workbooklib/excel_creation_utils.py	`65%`	❌	`99` `108-113` `118-125` `129-147` `160-164` `178-181`
census_historical_migration/workbooklib/federal_awards.py	`77%`	❌	`129-132` `360` `401-409` `419-444` `468-469` `689-790`
census_historical_migration/workbooklib/findings.py	`47%`	❌	`75-80` `148-168` `173-193` `210-212` `219-241`
census_historical_migration/workbooklib/findings_text.py	`83%`	❌	`47-48` `90-108`
census_historical_migration/workbooklib/notes_to_sefa.py	`66%`	❌	`34-38` `101-102` `104-105` `107-108` `136-143` `152-160` `168-170` `279-326`
census_historical_migration/workbooklib/post_upload_utils.py	`21%`	❌	`22-35` `66-83` `89-111`
census_historical_migration/workbooklib/secondary_auditors.py	`90%`	✅	`128-130` `162-180`
census_historical_migration/workbooklib/workbook_builder.py	`38%`	❌	`14-17` `26-41`
census_historical_migration/workbooklib/workbook_builder_loader.py	`41%`	❌	`18-30`
config/error_handlers.py	`94%`	✅	`22`
config/test_settings.py	`92%`	✅	`33-34` `49-50`
config/urls.py	`72%`	❌	`88`
dissemination/file_downloads.py	`81%`	❌	`43-61` `91-93`
dissemination/forms.py	`87%`	✅	`135` `144` `255`
dissemination/search.py	`61%`	❌	`101` `108` `110` `113-121`
dissemination/summary_reports.py	`78%`	❌	`274` `300-302` `306-310` `421` `438` `459` `511-575` `603` `638-640` `664-672`
dissemination/test_search.py	`93%`	✅	`51-66` `473-474` `579-596` `608-632` `644-669` `677-693`
dissemination/test_summary_reports.py	`98%`	✅
dissemination/views.py	`76%`	❌	`134` `140-142` `159-223` `266` `296` `298` `332` `383` `385` `387` `465-470`
dissemination/migrations/0002_general_fac_accepted_date.py	`47%`	❌	`10-12`
dissemination/searchlib/search_alns.py	`37%`	❌	`44-58` `78-110` `115-177` `184-187`
dissemination/searchlib/search_direct_funding.py	`86%`	✅	`21-22`
dissemination/searchlib/search_findings.py	`76%`	❌	`18-24` `34` `36` `38`
dissemination/searchlib/search_general.py	`96%`	✅	`138`
dissemination/searchlib/search_passthrough_name.py	`35%`	❌	`21-31`
djangooidc/backends.py	`78%`	❌	`32` `57-63`
djangooidc/exceptions.py	`66%`	❌	`19` `21` `23` `28`
djangooidc/oidc.py	`16%`	❌	`32-35` `45-51` `64-70` `92-149` `153-199` `203-226` `230-275` `280-281` `286`
djangooidc/views.py	`81%`	❌	`22` `43` `109-110` `117`
djangooidc/tests/common.py	`97%`	✅
report_submission/forms.py	`92%`	✅	`35`
report_submission/test_views.py	`98%`	✅	`796`
report_submission/views.py	`77%`	❌	`90` `235` `259-260` `265-266` `306-476` `479-489` `548` `584-586` `594-595` `598-600`
report_submission/templatetags/get_attr.py	`76%`	❌	`8` `11-14` `18`
support/admin.py	`88%`	✅	`76` `79` `84` `91-97` `100-102`
support/cog_over.py	`91%`	✅	`30-33` `93` `145`
support/test_admin_api.py	`81%`	❌	`23` `147-148` `237-238` `317-318`
support/test_cog_over.py	`98%`	✅	`134-135` `224`
support/management/commands/seed_cog_baseline.py	`98%`	✅	`20-21`
support/models/cog_over.py	`89%`	✅	`103-104`
tools/update_program_data.py	`89%`	✅	`96`
users/admin.py	`99%`	✅	`27`
users/auth.py	`96%`	✅	`58-59`
users/models.py	`96%`	✅	`18` `74-75`
users/fixtures/user_fixtures.py	`91%`	✅

Minimum allowed coverage is 85%

Generated by 🐒 cobertura-action against ecd9cfa

jperson1 · 2024-05-14T16:54:10Z

Testing this branch from GitHub Actions has proven difficult - to ensure this doesn't hang for too long, I've un-drafted it without the workflow changes. A later PR can pick up workflow changes to get a11y tests running on every PR.

backend/dissemination/management/commands/end_to_end_test_data_generator.py

danswick · 2024-05-17T22:13:13Z

backend/cypress/support/log-functions.js

Great addition!

danswick

One small thing we can tackle next week (happy to help): let's remove the old test_a11y commands and dependencies and do a sweep for straggler config or helper files previously used for pa11y and maybe lighthouse (though I'm not sure if there are other reasons to keep it around).

We should replace the old a11y npm commands with new ones for cypress. It might be useful to add a local-specific npm command that automatically loads fixtures as well.

danswick · 2024-05-17T22:22:28Z

docs/testing.md

Thank you for updating the docs! This is excellent.

danswick · 2024-05-17T22:23:38Z

backend/audit/fixtures/dissemination.py

Great idea 👍

danswick · 2024-05-17T22:25:28Z

This is really excellent, @jperson1! It looks fairly feature-complete to me, but I think we need to do a sweep for a couple pa11y stragglers just to make sure.

danswick

Thanks for the updates, @jperson1! For good measure, I ran the a11y tests manually and everything looks great here.

jperson1 added 12 commits May 9, 2024 10:02

A11y - Install Cypress Axe, new test file

ce0d153

A11y - Allow SAC/home access if DISABLE_AUTH =true

b1513f2

A11y - load_fixtures for new SACs, dummy dissem

e28d1fa

Linting, remove unused file.

9968813

A11y - HTML and SCSS changes for accessibility

92b0d3f

A11y - Docs update for new workflow

45870c3

Add accessibility tests to testing-from-ghcr.yml

45b190c

Cypress - Update report upload page input ids

376e7a7

Lint - Unused imports

f55afe7

Lint - Various black changes, not a11y related

04ee30c

Lint - Reformat new cypress files

6fee8c6

Merge branch 'main' into jp/3618-refactor-accessibility-testing

ab40b7b

jperson1 self-assigned this May 9, 2024

jperson1 temporarily deployed to dev May 9, 2024 14:47 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta May 9, 2024 14:47 — with GitHub Actions Inactive

Re-linting - Whoops, wrong version of black

d3dc067

jperson1 temporarily deployed to dev May 9, 2024 14:54 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta May 9, 2024 14:54 — with GitHub Actions Inactive

GHA - upload screenshots, use dc-web, spaces

1d00abb

jperson1 temporarily deployed to dev May 9, 2024 15:41 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta May 9, 2024 15:41 — with GitHub Actions Inactive

Testing with just a11y workflow

85a767f

jperson1 temporarily deployed to dev May 9, 2024 17:35 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta May 9, 2024 17:35 — with GitHub Actions Inactive

Testing with a historical data load

ffbbfbf

jperson1 temporarily deployed to dev May 9, 2024 17:41 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta May 9, 2024 17:41 — with GitHub Actions Inactive

Add npm i to testing doc.

5e58dff

jperson1 temporarily deployed to dev May 17, 2024 16:42 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta May 17, 2024 16:42 — with GitHub Actions Inactive

phildominguez-gsa reviewed May 17, 2024

View reviewed changes

backend/dissemination/management/commands/end_to_end_test_data_generator.py Outdated Show resolved Hide resolved

Strike "TESTING" from e2e TSTDAT generator command

a736e10

jperson1 temporarily deployed to dev May 17, 2024 17:59 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta May 17, 2024 17:59 — with GitHub Actions Inactive

phildominguez-gsa previously approved these changes May 17, 2024

View reviewed changes

danswick reviewed May 17, 2024

View reviewed changes

backend/cypress/support/log-functions.js Outdated

Copy link

Contributor

danswick May 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great addition!

jperson1 reacted with heart emoji

danswick requested changes May 17, 2024

View reviewed changes

danswick reviewed May 17, 2024

View reviewed changes

docs/testing.md Outdated

Copy link

Contributor

danswick May 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for updating the docs! This is excellent.

danswick reviewed May 17, 2024

View reviewed changes

backend/audit/fixtures/dissemination.py Outdated

Copy link

Contributor

danswick May 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great idea 👍

jperson1 added 4 commits May 20, 2024 12:13

Remove references to pa11y

33ef01d

Remove references to lighthouse

4b063e6

Include an a11y command in package.json

a4cb578

Remove a11y reference from staging deploy yml

adc8770

jperson1 dismissed phildominguez-gsa’s stale review via adc8770 May 20, 2024 16:30

danswick previously approved these changes May 22, 2024

View reviewed changes

danswick mentioned this pull request May 22, 2024

Re-enable automated a11y testing #3879

Open

Merge branch 'main' into jp/3618-refactor-accessibility-testing

ecd9cfa

jperson1 dismissed danswick’s stale review via ecd9cfa May 23, 2024 15:01

jperson1 temporarily deployed to dev May 23, 2024 15:01 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta May 23, 2024 15:01 — with GitHub Actions Inactive

danswick approved these changes May 23, 2024

View reviewed changes

jperson1 added this pull request to the merge queue May 23, 2024

Merged via the queue into main with commit d3c61a2 May 23, 2024
12 checks passed

jperson1 deleted the jp/3618-refactor-accessibility-testing branch May 23, 2024 17:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor Accessibility Testing #3814

Refactor Accessibility Testing #3814

jperson1 commented May 9, 2024 •

edited

github-actions bot commented May 9, 2024 •

edited

github-actions bot commented May 9, 2024 •

edited

github-actions bot commented May 9, 2024 •

edited

jperson1 commented May 14, 2024

danswick May 17, 2024

danswick left a comment

danswick May 17, 2024

danswick May 17, 2024

danswick commented May 17, 2024

danswick left a comment

Refactor Accessibility Testing #3814

Refactor Accessibility Testing #3814

Conversation

jperson1 commented May 9, 2024 • edited

Refactor Accessibility Testing

Brief overview:

Changes:

How to test:

All the accessibility changes by page:

Some funky fresh screenshots:

PR checklist: submitters

PR checklist: reviewers

github-actions bot commented May 9, 2024 • edited

github-actions bot commented May 9, 2024 • edited

github-actions bot commented May 9, 2024 • edited

jperson1 commented May 14, 2024

danswick May 17, 2024

Choose a reason for hiding this comment

danswick left a comment

Choose a reason for hiding this comment

danswick May 17, 2024

Choose a reason for hiding this comment

danswick May 17, 2024

Choose a reason for hiding this comment

danswick commented May 17, 2024

danswick left a comment

Choose a reason for hiding this comment

jperson1 commented May 9, 2024 •

edited

github-actions bot commented May 9, 2024 •

edited

github-actions bot commented May 9, 2024 •

edited

github-actions bot commented May 9, 2024 •

edited