pa11y reports an axe warning as an error

[rootwork]

Expected behaviour

When an axe rule can't determine something, it should be reported as a warning, not an error. If that sounds vague, it's because I'm trying to extrapolate from the rule on which I see this happening to generalize about any rule in which axe reports x could not be determined due to y.

The specific rule in this case is color-contrast. You'll see toward the bottom of that definition page it says:

This rule will not report on text elements that have a background-image, are obscured by other elements or are images of text.

For instance, if an element has a gradient background (which is a type of background-image in CSS) overlaid with text, looking at it using the axe browser extension it will say:

Element's background color could not be determined due to a background gradient.

When pa11y receives a notice like this, it should consider it a warning, not an error.

Actual behaviour

Instances like this are reported as errors:

 • Error: Elements must have sufficient color contrast (https://dequeuniversity.com/rules/axe/4.4/color-contrast?application=axeAPI)

Steps to reproduce

Unfortunately running pa11y (or axe-core/cli) against a CodePen doesn't really work, but here's some really simple code to test:
https://codepen.io/rootwork/pen/zYPwQbv

If you inspect that using axe's browser extension, you'll see the warning noted above.

In my case I was running pa11y locally using pa11y --runner axe http://localhost:9050/ against a page that had similar a text-on-gradient section.

Finally, if I'm wrong about diagnosing this issue and it's actually a problem with axe-core/cli, please accept my apologies and I'll re-report it over there.

Environment:

  System:
    OS: Linux 5.4 Linux Mint 20.3 (Una)
    CPU: (4) x64 Intel(R) Core(TM) i5-6600 CPU @ 3.30GHz
    Memory: 1.13 GB / 23.35 GB
    Shell: 5.8 - /usr/bin/zsh
  Binaries:
    Node: 16.13.2 - ~/n/bin/node
    Yarn: 1.22.10 - /usr/local/bin/yarn
    npm: 8.1.2 - ~/n/bin/npm
  pa11y: 6.1.1

[dboulet]

I’m seeing this same issue, the problem is that pa11y is reporting “review items” from axe as errors. From the Axe documentation, we can see that axe returns two types of issues, violations and incomplete. From the documentation:

The incomplete array (also referred to as the "review items") indicates which nodes could neither be determined to definitively pass or definitively fail. They are separated out in order that a user interface can display these to the user for manual review (hence the term "review items").

Looking at the code in /lib/runners/axe.js, the incomplete items are processed by the processIncomplete() function:

pa11y/lib/runners/axe.js

Lines 147 to 150 in 9cf10d4

	function processIncomplete(issue) {
	issue.type = 'warning';
	return processIssue(issue);
	}

Within the function, the issue type property is set to warning and then the data is further processed using the processIssue() function before being returned. The problem is that the processIssue() function completely ignores the existing type property set by processIncomplete() and instead determines a new type based on the impact property.

pa11y/lib/runners/axe.js

Lines 168 to 169 in 9cf10d4

	return elements.map(element => ({
	type: axeImpactToPa11yLevel(axeIssue.impact),

This will cause the issue to have a type of error if the impact is critical or serious, completely ignoring the fact that axe reported the issue as incomplete.

This should probably be changed so that incomplete issues from axe can at most have a level of warning or notice.

[daemoncron]

Hey I know this is old, am I missing something here or is this just a one-line fix?

pa11y/lib/runners/axe.js

Line 169 in 9cf10d4

type: axeImpactToPa11yLevel(axeIssue.impact),

type: axeImpactToPa11yLevel(axeIssue.type || axeIssue.impact),
or even just
type: axeImpactToPa11yLevel(axeIssue.type),?

[danyalaytekin]

Thanks @rootwork for your fantastic description and @dboulet @daemoncron for zeroing in on the issue. Looking into this.