Accessibility Features Proposal - STF 2

[calebfoss]

Topic

Written for the p5.js Documentation Organization & Accessibility project, this issue proposes a plan for upcoming improvements to p5's accessibility features.

Background on Existing Features

p5's gridOutput() and textOutput() methods are very cleverly designed tools that aim to automatically describe the content on the canvas.

Current limitations and issues for these methods include:

• descriptions are exclusively in English
• text is not described
• 3D shapes are not described
• 2D primitives rendered in webGL are described incorrectly due to not factoring in the camera's perspective
• 2D primitives with similar features are combined in descriptions, resulting in inaccurate counts of shapes
• 2D primitives positioned outside the canvas are described as though they are visible
• Custom 2D shapes made with beginShape() etc. are not described

So currently, these functions can accurately describe sketches featuring exclusively 2D primitives, each with somewhat distinct features, all on the canvas for an English-reading audience. They describe the color, approximate location, and approximate percentage of space occupied of each shape.

I would argue that this improves accessibility for a very limited selection of p5 sketches. Even those that can be accurately described.

Example

textOutput() produces the following description

Your output is a, 400 by 400 pixels, white canvas containing the following 4 shapes:

    yellow circle, at middle, covering 79% of the canvas.
    black circle, at top left, covering 5% of the canvas.
    black circle, at top right, covering 5% of the canvas.
    black arc, at middle, covering 22% of the canvas.

yellow circle	location = middle	area = 79%
black circle	location = top left	area = 5%
black circle	location = top right	area = 5%
black arc	location = middle	area = 22%

The description is technically accurate, but I would argue that it is significantly less useful at conveying the content than a concise, manually written description such as

A yellow smiley face on a light gray background.

Side Note on AI

An idea I anticipate coming up in relation to the issue I presented above is to use AI-generated descriptions. I would like to take a moment to strongly advise against that because

• A feature like that would most likely rely on a Large Language Model, which could put p5 in a position of depending on exploitative and inhuman labor practices.
• Even the most cutting edge AI models have an ongoing issue with conveying false information as fact (often referred to as "hallucinating").
• AI models have a well-documented habit of generating harmful content, particularly directed toward people of marginalized identities.

Moving forward

I would say that these features have a fork in the road ahead. They can either be further developed to improve accuracy and warn the creator about limitations or they can be removed from p5. With respect for the contributors who developed these very impressive features, unfortunately, I think the best path forward is the latter.

Keeping textOutput() and gridOutput()

That being said, if textOutput() and gridOutput() not removed, I would recommend:

• Throw a friendly error if these functions are used in webGL mode. I think accurately describing 3D content is an extremely ambitious goal. Accounting for camera perspective, occlusion, and lighting would be an enormous challenge. Describing custom models and procedurally geometry would be perhaps an even bigger challenge. Note: I had offered to implement the webGL error message a while back, started to work on it, got stuck on how to work it into the FES, and dropped the ball.
• Implement descriptions for text drawn on the canvas in 2D
• Update documentation in textOutput() and gridOutput() to clearly indicate their limitations
• Update outputs.js to check if a shape is off canvas and remove it from "ingredients" if so. This would involve collision detection, so this could be developed in parallel with a series of methods to help creators handle collision.
• Add translations and add a parameter to these methods to allow creators to select a language.

My opinion is that the above changes are a lot of work for features that would only improve accessibility in very limited situations.

Removing textOutput() and gridOutput()

Instead, I would recommend removing those methods and focusing on encouraging creators to describe their sketches effectively and to provide good resources to do so. While automating this task is an enticing idea, there is something very human about effectively conveying visual content through written language. Learning to write a helpful accessible description is just as valuable as to making strong web content as learning to generate imagery programmatically.

Already, the Writing Accessible Canvas Descriptions tutorial is a fantastic resource. Additionally, sketches featured Examples page on the new p5 site include describe(), describeElement(), textOutput(), and/or gridOutput().

Returning an Object with a Describe Method

Here is an idea that would be a big undertaking but perhaps not as big as the improvements textOutput() and gridOutput() listed above and, I would argue, would have a much bigger impact:

What if instead of returning the p5 instance, visual functions returned an object that contained information about it and a method to describe it?

Example:

let sunY = 50;

function setup() {
  createCanvas(400, 400);
  describe('An outdoor scene.');
}

function draw() {
  background('lightblue');

  fill('yellow');
  const sun = circle(50, sunY, 50);

  fill('green');
  const grass = rect(0, 300, width, 100);
  grass.describe('An empty grassy space is in the foreground.');

  if (sun.y < 200) {
    sun.describe('The sun is high in the sky.');
  } else if (sun.y < grass.y + sun.diameter / 2) {
    sun.describe('The sun is setting at the horizon.');
  }
  sunY++;
}

This is essentially a way to more directly connect what describeElement() does to the functions producing what it describes. Storing the arguments passed into the shape functions helps the creator keep track of what those values represent.

In many cases, an "element" of the composition will be rendered using multiple functions, so describeElement() will still be useful for those cases.

A potential additional feature this could open the door for would be using isPointInPath() and/or isPointInStroke() to store in this object a boolean property indicating whether the shape was clicked this frame.

In my experience, the work required to check if a shape was clicked (either manually through collision detection algorithms or through loading a library) is a significant obstacle to p5 creators.

Performance impact would need testing, as the way I am describing this feature, those methods would need to be called every time any shape is drawn. Additionally the mouse coordinates would need to be transformed using the current transformation matrix.

I believe that some of the existing helper functions for textOutput() and gridOutput() could be repurposed to keep track of "ingredients" and reuse HTML elements.

Color Contrast

Issue #6971 proposes a way to check whether the contrast between colors in a sketch is meeting WCAG AA or AAA.

I drafted a method to calculate contrast based on WebAIM's tool. When passed two p5.Color's, it returns the contrast ratio between them: https://editor.p5js.org/cfoss/sketches/-okUAfg7Z

Note that this does not currently account for alpha level or blend mode, and it would need to do that for accuracy.

This could be modified to check for specific AA and AAA target ratios. I would argue, though, that this information would be better placed in the reference, where it could be linked, contextualized, and updated without modifying the function itself.

@limzykenneth responded

I wonder if this can be expanded into a wider accessibility checker feature that can form part of the Web Accessibility module of p5.js? Eg. also adding a function that when passed a HTML container element, check everything inside it for adherene to best practices of web accessibilty/WCAG recommendations?

While I think a broader accessibility checker is an exciting idea, I believe that implementing it such that it reliably gives accurate results would be a daunting task.

I am not aware of a way to check what default styles a browser has applied to an element beyond the styles set via CSS, style property, etc. On top of that, accounting for all the possible factors that would influence which element appears on top of which would be very challenging.

For those reasons, when it comes to color contrast at least, I do not think automatically checking elements is a realistic goal.

I could imagine an automatic checker for semantic structure: checking for more than one h1, mismatching h1 and title, disorganized heading levels, etc. Having a button in the p5 editor that opens such a checker would be great.

Flashing light

WCAG 2.3.2 specifies limits on flashing light.

I would propose a method for testing whether what is rendered to the canvas meets these limits written something like

function setup() {
  createCanvas(400, 400);
  checkFlashing();
}

that would throw a friendly error if and when the sketch fails the test. This could share a helper function with the contrast function as both are based on relative luminance.

Calculating relative luminance across the canvas every frame would likely be quite computationally expensive, but for running a test, I think that would be acceptable.

Disabling Animation

WCAG 2.3.3 specifies that "Motion animation triggered by interaction can be disabled, unless the animation is essential to the functionality or the information being conveyed."

I would propose that, by default, p5 create an HTML element that acts as a pause/play button for the sketch, connected to loop()/noLoop(). This behavior could be disabled or reenabled with methods called something like noPause() and allowPause() respectively.

[Qianqianye]

Thanks @calebfoss! Tagging @limzykenneth and @davepagurek to review.