rules option ignored

[edsnowden]

This is surely a misunderstanding or error on my part, but as far as I can tell the 'rules' option is checked but ignored.

Expected behaviour

When a list of rules is given in the rules option, only those rules are checked for/reported.

Actual behaviour

Specified rules must be valid, but appear to have no effect.

Steps to reproduce

Install node and pa11y

mkdir node_modules
npm install node
npm install pa11y
PATH="node_modules/.bin:$PATH"

Create a file bug.js:

const pa11y = require('pa11y')

async function runTests (options) {
    try {
        let results = await pa11y("http://lwn.net", options);
        console.log(results.issues.length, options);
        console.log(results);
    } catch (e) {
        console.log(e.message, options);
    }
};

async function main () {
    console.log(pa11y.defaults.userAgent);
    console.log("node", process.version);

    await runTests({
        rules: [
            "ThisIsNotAValidRule"
        ]
    });

    await runTests({});

    await runTests({
        rules: [
            "Principle1.Guideline1_4.1_4_3_Contrast",
        ]
    });
}

main();

Run with

node bug.js

Observe the output is the same whether the rules are specified or not, and the errors do not relate to the specified rule:

pa11y/6.2.3
node v19.8.1
Evaluation failed: Error: ThisIsNotAValidRule is not a valid WCAG 2.1 rule
    at configureHtmlCodeSniffer (__puppeteer_evaluation_script__:67:11)
    at Object.window.__pa11y.runners.htmlcs (__puppeteer_evaluation_script__:49:2)
    at Object.runPa11y [as run] (__puppeteer_evaluation_script__:47:52) { rules: [ 'ThisIsNotAValidRule' ] }
2 {}
{
  documentTitle: 'Welcome to LWN.net [LWN.net]',
  pageUrl: 'https://lwn.net/',
  issues: [
    {
      code: 'WCAG2AA.Principle1.Guideline1_3.1_3_1.H49.Tt',
      type: 'error',
      typeCode: 1,
      message: 'Presentational markup used that has become obsolete in HTML5.',
      context: '<tt>perf</tt>',
      selector: 'html > body > div:nth-child(5) > div:nth-child(1) > div:nth-child(2) > div > div:nth-child(1) > div:nth-child(21) > p:nth-child(2) > tt',
      runner: 'htmlcs',
      runnerExtras: {}
    },
    {
      code: 'WCAG2AA.Principle1.Guideline1_3.1_3_1.H49.Center',
      type: 'error',
      typeCode: 1,
      message: 'Presentational markup used that has become obsolete in HTML5.',
      context: '<center>\n            <p>\n            <s...</center>',
      selector: 'html > body > center',
      runner: 'htmlcs',
      runnerExtras: {}
    }
  ]
}
2 { rules: [ 'Principle1.Guideline1_4.1_4_3_Contrast' ] }
{
  documentTitle: 'Welcome to LWN.net [LWN.net]',
  pageUrl: 'https://lwn.net/',
  issues: [
    {
      code: 'WCAG2AA.Principle1.Guideline1_3.1_3_1.H49.Tt',
      type: 'error',
      typeCode: 1,
      message: 'Presentational markup used that has become obsolete in HTML5.',
      context: '<tt>perf</tt>',
      selector: 'html > body > div:nth-child(5) > div:nth-child(1) > div:nth-child(2) > div > div:nth-child(1) > div:nth-child(21) > p:nth-child(2) > tt',
      runner: 'htmlcs',
      runnerExtras: {}
    },
    {
      code: 'WCAG2AA.Principle1.Guideline1_3.1_3_1.H49.Center',
      type: 'error',
      typeCode: 1,
      message: 'Presentational markup used that has become obsolete in HTML5.',
      context: '<center>\n            <p>\n            <s...</center>',
      selector: 'html > body > center',
      runner: 'htmlcs',
      runnerExtras: {}
    }
  ]
}

Environment

  System:
    OS: Linux 4.18 CentOS Stream 8
    CPU: (72) x64 Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
    Memory: 174.99 GB / 188.80 GB
    Shell: 4.4.20 - /usr/local/bin/bash
  Binaries:
    Node: 19.8.1 - node_modules/.bin/node
    npm: 6.14.10 - /usr/bin/npm
  pa11y: 6.2.3

[danyalaytekin]

Hi @edsnowden, thanks for this report, and for your code and logs which made it straightforward to understand the issue and reproduce the behaviour. Also, that is a very pleasant amount of memory.

For HTML_CodeSniffer: I'm reasonably confident this is the currently intended behaviour of options.rules, going by the name of its non-obvious CLI counterpart, --add-rule, and by this test's name. I agree that rules is not a very descriptive name though, if I'm understanding this correctly, so maybe we should improve how we explain their equivalence.

For axe: --add-rule and options.rules are currently documented as being only for HTML_CodeSniffer, but I think we use this property internally for a different purpose for axe, where the intended behaviour appears to be 'set' rather than 'add', according to the name of the test I just linked, but I'm actually seeing the 'add' behaviour locally for axe too.

Are you comfortable with this behaviour, with the understanding it represents addition rather than replacement? Some other changes we could make:

1. (breaking) simply rename rules to be clearer; probably not a good idea since it would inconvenience current users of the property
2. (minor) same as (1), but also leave rules (now an alias for that new property) in place
3. (minor) explain this better in our documentation, whether in code or markdown. When you first learned about options.rules, where were you looking? We could try to clarify this to readers there