First stab at making Mudlet's text window readable by screenreaders #5985

bauermann · 2022-02-23T19:47:46Z

Hello! This is what I have so far regarding fixing issue #3342. It’s not done yet because there’s a significant issue I’m still trying to figure out how to fix (ideas welcome):

When new text arrives in the TTextEdit, the screen reader doesn’t announce it even though the widget is sending the ‘TextInsertEvent’ with the new text. The only time it works is when Caret Mode is enabled (press F7 to activate it), so what I believe is going on is that the screen reader only pays attention to the emitted ‘TextInsertEvent’ when the TTextEdit is focused.

Therefore I’m trying to figure out a way to tell the screen reader that changes in TTextEdit should be announced even when it’s not focused. Orca’s documentation mentions something called “live regions” which seems to be what I need, but so far I have the impression that it’s a concept applied only to web content. I hope I’m wrong though.

Using Orca’s “Speak entire document” command (KP_PLUS or + Semicolon) works though, and so does speaking when moving the cursor in Caret Mode which means that the widget’s contents are being exposed correctly to the screen reader. It’s just that there’s something more I need to do in order to make the screen reader know that it needs to announce new text as it arrives.

Nobody asked to implement Caret Mode but I did anyway, inspired by the feature of the same name in Firefox. It seems useful to me, please let me know what you think.

I dropped the commit where I was removing the HTML markup from the tooltips in TConsole’s buttons. The screen reader still speaks them so it’s a bit annoying, but now there’s only <p>…</p> markup whereas before it was <html><head/><body><p>…</p></body></html>, so a lot better now.

Also, since Qt does expect HTML markup in the tooltip then it should strip the
markup before passing the tooltip contents along to the screen reader. So it’s a
problem to be fixed somewhere else in the technology stack.

I’m testing with GNOME on Linux, with the Orca screen reader.

I’d appreciate more testing, as well as feedback on the code and design.

I don’t expect this PR to be merged. It’s mostly so that the CI can generate snapshot builds.

add-deployment-links · 2022-02-23T19:48:08Z

Hey there! Thanks for helping Mudlet improve. 🌟

Test versions

You can directly test the changes here:

No need to install anything - just unzip and run.
Let us know if it works well, and if it doesn't, please give details.

github-actions · 2022-02-23T19:48:21Z

	Fails
🚫	Source file src/TAccessibleTextEdit.cpp includes a TODO with no Mudlet issue link. New TODO items in source files must have an accompanying github issue
🚫	PR title must start with `fix`, `improve`, `add` or `infra` for release notes purposes.

Generated by 🚫 dangerJS against e673121

github-actions

clang-tidy made some suggestions

src/TAccessibleTextEdit.cpp

src/TTextEdit.cpp

src/mudlet.cpp

vadi2 · 2022-02-23T20:22:32Z

Nice! Thanks for this. Will review it in detail in the coming days. In the meantime I posted it on our Discord's accessibility channel for people to start testing.

bauermann · 2022-02-23T22:31:06Z

Em quarta-feira, 23 de fevereiro de 2022, às 17:22:43 -03, Vadim Peretokin escreveu:

Nice! Thanks for this. Will review it in detail in the coming days. In the meantime I posted it on our Discord's accessibility channel for people to start testing.

That’s great! Thank you.

SlySven

Just an initial review...!

src/TAccessibleTextEdit.h

src/CMakeLists.txt

src/TConsole.cpp

src/TTextEdit.cpp

SlySven · 2022-02-24T17:04:10Z

src/TTextEdit.cpp

@@ -602,7 +640,8 @@ int TTextEdit::drawGraphemeBackground(QPainter& painter, QVector<QColor>& fgColo
    }
    textRects.append(textRect);
    QColor bgColor;
-    if (Q_UNLIKELY(static_cast<bool>(attributes & TChar::Reverse) != charStyle.isSelected())) {
+    bool caretIsHere = mudlet::self()->isCaretModeEnabled() && mCaretLine == line && mCaretColumn == column;
+    if (Q_UNLIKELY(static_cast<bool>(attributes & TChar::Reverse) != (charStyle.isSelected() != caretIsHere))) {


I'm not sure of this - originally it was to EXCLUSIVE-OR the reverse text mode with the reverse effect that selecting text does (so when combined the state is the same as if neither was applied) - what does the "CaratMod" mode do to how the text is/should be shown?

This is to implement the caret mode’s cursor. It is displayed as a reversed-color character cell, similar to old terminals.

The change above reverses the color when the character cell has normal color, and uses normal color when it has reversed-color.

So logically you have to handle all three conditions at the same time and apply the reverse effect when an odd number of them are true. Maybe the logic you have chosen will work, I'll have to think about it a bit more.

src/TTextEdit.cpp

src/main.cpp

SlySven · 2022-02-24T17:20:13Z

src/mudlet.cpp

@@ -2976,6 +2977,12 @@ void mudlet::slot_multi_view(const bool state)
    }
 }

+void mudlet::slot_caret_mode(bool state)
+{
+    // FIXME: Support multiple hosts.


Yeah, you will need to handle multiple host instances - and using // FIXME: is not making this PR "dangerous" (upsetting the Danger Bot that watches some aspects of PRs!) whereas using // TODO: definitely would... 😲

I plan to implement this before making the definitive PR. This comment is just so that I don’t forget about it. :)

Hadi-1624 · 2022-02-25T10:01:04Z

I believe this is not working in windows at all.
Testing with NVDA screen reader on windows 10. when mudlet connects to a mud, the keyboard focus is on the input field and will refuse to go anywhere else, even if I press the f7 key.

Just to make sure, here's how I imagine the client could be accessible:

screen reader reads incoming text in the output window
Mudlet by default has the keyboard focus on the input window. there should be an implementation in which the keyboard focus and cursor moves from the input to the output window, so that the output window could be navigated by the arrow keys and be read. this preferably should be switched by a hotkey so it could be fast moving between the input and output windows.
Functions that sends the last 10 lines in the output window to the screen reader, which then could be assigned to hotkeys, so pressing control+shift+5 for example, would send the fifth line from bottom to the scren reader to be read.

ironcross32 · 2022-02-25T16:22:26Z

Hi, the F7 mode isn't activating in this case. I can verify that a connection has been made to the MUD I intended by running OCR on the game window. I am testing on Windows 10.

bauermann · 2022-02-26T18:15:09Z

Hello Hadi, Thank you very much for testing! It’s very helpful. I’m sorry that you had such a bad experience. I thought that there could be issues on Windows, but I had hope it wouldn’t be as bad as it it. :( Em sexta-feira, 25 de fevereiro de 2022, às 07:01:15 -03, Hadi Rezaei escreveu:

I believe this is not working in windows at all.

Ouch. I’m sorry about that. I don’t have access to a Windows machine, but I will do some research and see if I can find something out. Perhaps even try it on Wine (a Windows emulator for Unix).

Testing with NVDA screen reader on windows 10. when mudlet connects to a mud, the keyboard focus is on the input field and will refuse to go anywhere else, even if I press the f7 key.

I’m surprised that F7 isn’t working. I’m just using a conventional Qt API to add that hot key. I’ll dig into it. It’s also possible to enable caret mode using the menu item “Caret Mode” under the ”Options” menu. It should have exactly the same effect as the F7 key. Would you mind checking whether it works that way?

Just to make sure, here's how I imagine the client could be accessible:

Thank you for this detailed explanation. It’s very helpful.

1. screen reader reads incoming text in the output window

I agree. This isn’t working even on my machine and it’s the most glaring and embarassing problem with my branch.

2. Mudlet by default has the keyboard focus on the input window. there should be an implementation in which the keyboard focus and cursor moves from the input to the output window, so that the output window could be navigated by the arrow keys and be read. this preferably should be switched by a hotkey so it could be fast moving between the input and output windows.

This is what I’m implementing with Caret Mode. It’s unfortunate that it isn’t working on Windows. I’ll try to fix it.

3. Functions that sends the last 10 lines in the output window to the screen reader, which then could be assigned to hotkeys, so pressing control+shift+5 for example, would send the fifth line from bottom to the scren reader to be read.

This is a great idea! Shouldn’t be hard to implement (famous last words). I’ll work on it.

bauermann · 2022-02-26T18:18:01Z

Hello ironcross32, Thank you for testing! I’m sorry that my branch failed so miserably. I was expecting problems, but thought it wouldn’t be this bad. Em sexta-feira, 25 de fevereiro de 2022, às 13:22:38 -03, ironcross32 escreveu:

Hi, the F7 mode isn't activating in this case. I can verify that a connection has been made to the MUD I intended by running OCR on the game window. I am testing on Windows 10.

I’m sorry about that. I’m really suprised by this problem, since I implemented the F7 hot key using a standard Qt API... It’s also possible to enable caret mode using the menu item “Caret Mode” under the ”Options” menu. It should have exactly the same effect as the F7 key. Would you mind checking whether it works that way?

mohammedradwan2003 · 2022-02-26T20:22:12Z

hi,
i totally agree with what was said, however let me clarify something
what they ment, a window which contains all text received by the mud, and input window contains all the commands i send
making it easier for us that way, both are separated and can be reached via a hot key
i hope this gets implimented
thanks
mohammed
can't wait to see this huge project coming out

ironcross32 · 2022-02-26T20:40:01Z

I've posted some messages in the Mudlet Discord under the Mudlet development channel that may be of assistance.

First, rename mLastRenderBottom to mLastRenderedOffset. The old name is misleading because it actually holds the line that was at the top of the screen when it was last rendered, not the bottom. Also add explanation of what mScrollVector means. These changes help understand the logic at TTextEdit::drawForeground().

bauermann · 2022-02-26T21:53:01Z

Hello Mohammed, Em sábado, 26 de fevereiro de 2022, às 17:22:22 -03, mohammedradwan2003 escreveu:

i totally agree with what was said, however let me clarify something what they ment, a window which contains all text received by the mud, and input window contains all the commands i send making it easier for us that way, both are separated and can be reached via a hot key i hope this gets implimented

Thank you for explaining your idea. So you mean that Mudlet should have an accessibility mode where there are two separate windows, one for input and one for output? That is is interesting, though if I understand correctly it would be out of scope for this specific issue (unless someone from the Mudlet project corrects me and says that it is in scope) so I cannot say that I would be able to implement it. I think it’s worth opening a new GitHub issue about it so that there can be a discussion focused specifically on your idea.

can't wait to see this huge project coming out

Me too! And it’s been a larger project than I originally thought it would be. :)

bauermann · 2022-02-26T21:55:05Z

Em sábado, 26 de fevereiro de 2022, às 17:40:12 -03, ironcross32 escreveu:

I've posted some messages in the Mudlet Discord under the Mudlet development channel that may be of assistance.

Thank you, I just saw it.

Hadi-1624 · 2022-02-26T22:09:04Z

Hi there bauermann
Thanks for working on accessibility of the client!

I tried to follow your suggestion and enable Caret mode from the options menu but i've faced another problem where, after connecting to a game, pressing alt will not bring the menus up. i've tried pressing alt, pressing it again and pressing down arrow but no luck.
Is there a way to open the option menu by a hotkey?
I do confirm that mudlet default hotkeys work however. package manager and module manager for example, can be opnned by their shortcuts.

bauermann · 2022-02-26T23:12:57Z

Hello Hadi, Em sábado, 26 de fevereiro de 2022, às 19:09:15 -03, Hadi Rezaei escreveu:

Thanks for working on accessibility of the client!

You’re welcome! I hope I can bring it to a usable state.

I tried to follow your suggestion and enable Caret mode from the options menu but i've faced another problem where, after connecting to a game, pressing alt will not bring the menus up. i've tried pressing alt, pressing it again and pressing down arrow but no luck. Is there a way to open the option menu by a hotkey? I do confirm that mudlet default hotkeys work however. package manager and module manager for example, can be opnned by their shortcuts.

I have the same results in Linux. I was also not able to access the menu just using keyboard input, and had the same behavior as you describe (including being able to bringing up the package manager). I’ll look into this issue as well.

bauermann · 2022-02-26T23:25:35Z

Here are ironcross32’s comments on Discord. They allowed me to post here:

Currently, I am not able to test the latest PR because it seems to be for Linux users only and I am on Windows. But reading the synopsis of the PR, I have a few concerns. The first of which is that while a Firefox style caret browsing mode sounds great in theory, it would have to stop the output from scrolling while it is active.

Indeed. And it does stop scrolling. The good news is that when caret mode is enabled, the output widget has focus and thus the screen reader reads the new text as it arrives at the bottom.

Second is the keystroke used to access it takes one hand off the keyboard long enough to tap it. something like tab or control + tab would work better in my opinion.

Good point. tab is already used for completion, but I think I can change the shortcut to control+tab. I will do that.

The third thing is that to make the screen reader see all incoming text could, in theory, be easily done by using something like https://github.com/dkager/tolk. Though I believe this would only work under Windows, unfortunately.

Indeed, unfortunately tolk is a Windows-only solution from what I can see. Theoretically, Qt should be able to provide cross-platform screen reader functionality though, and tolk or something like it shouldn’t be necessary. I must be doing something wrong in my code.

It seems as if from the little bit of digging around that I've done, not actually being able to use Mudlet, that it supports plugins or packages. So, if a plugin implemented a catch-all trigger which sent all of the text contained within that trigger to the screen reader using Tolk, that would be sufficient for us to start using it.

IMHO it’s too early to give up on the current approach, as Qt theoretically should work. If all else fails though, then perhaps this could be considered (though I wouldn’t be the one implementing it since I don’t develop on Windows).

As a P.S. the author of the PR was correct that live regions are for web content, and wouldn't be useful in making the screen reader detect incoming text.

Thank you! This is very useful information. If someone knows of a feature equivalent to live regions but for desktop apps, I’d love to know about it.

ironcross32 · 2022-02-27T01:31:46Z

I'm 85% sure that QT will not do what you want. QT does have an accessibility layer, but it is for the controls. All of the controls unrelated to the actual game window do already seem to work. What needs to happen here is that text needs to be captured and sent to the running screen reader through its API. Unfortunately, there is no solution for this that fits all applications due to working cross-platform, and every screen reader has its own API. So there would need to be contingencies for sending through speech dispatcher in Linux and either writing your own class or using Tolk or Universal Speech on Windows. I have no clue about Mac.

This should be the first priority, because anything else is unimportant until incoming text is automatically read. Maybe if you can get this working under Linux, someone else can come along and duplicate it for Windows.

bauermann · 2022-02-27T03:37:03Z

Em sábado, 26 de fevereiro de 2022, às 22:31:58 -03, ironcross32 escreveu:

I'm 85% sure that QT will not do what you want. QT does have an accessibility layer, but it is for the controls. All of the controls unrelated to the actual game window do already seem to work. What needs to happen here is that text needs to be captured and sent to the running screen reader through its API. Unfortunately, there is no solution for this that fits all applications due to working cross-platform, and every screen reader has its own API. So there would need to be contingencies for sending through speech dispatcher in Linux and either writing your own class or using Tolk or Universal Speech on Windows. I have no clue about Mac.

Qt has an API to send arbitrary text to be read out lout in a platform independent way: https://doc.qt.io/qt-5/qtexttospeech.html#say I was hoping there’s a higher-level feature analogous to live regions but for desktop apps which is more standard accross accessibility technologies and platforms. If that’s not the case, then I can implement something directly using QTextToSpeech::say().

This should be the first priority, because anything else is unimportant until incoming text is automatically read.

I agree.

Maybe if you can get this working under Linux, someone else can come

along and duplicate it for Windows. I hope it doesn’t come to that, but it’s a last resort possibility.

The current cursor position isn't relevant in the calculation.

This fixes the following warning: 'TAccessibleTextEdit::text' hides overloaded virtual function Adjust lineForOffset() and cursorPosition() to take into account the newlines added by join(). Also, simplify text(int startOffset, int endOffset) using the new text(QAccessible::Text t) method.

… on the last char

Support only selection 0.

…and textAtOffset()

It's partially based on QAccessibleTextWidget::attributes() from Qt 5.15.

Also, remove ‘DocumentContentChanged’ event from TConsole::print() methods because both call TTextEdit::showNewLines(), which already sends ‘TextInsertEvent’.

Thanks to Stephen Lyons for the suggestion.

bauermann · 2022-02-27T05:45:46Z

Just an initial review...!

Thank you very much for this detailed review! I just force-pushed my branch with the changes corresponding to the resolved comments. I still have to work on the unresolved ones.

vadi2 · 2022-02-27T08:06:22Z

It is far too early to give up on the current cross-platform approach. We've discussed the reasons for this approach previously in #3342, let's not get sidetracked here again as well.

…ng-tidy Example warning: warning: variable lengthSoFar is not initialized [cppcoreguidelines-init-variables] int lengthSoFar; ^ = 0

Example warning: warning: Potential memory leak [clang-analyzer-cplusplus.NewDeleteLeaks] ⋮ src/TTextEdit.cpp:351:42: note: Memory is allocated QAccessible::updateAccessibility(new QAccessibleTextInsertEvent(this, ^ src/TTextEdit.cpp:355:1: note: Potential memory leak

src/TAccessibleTextEdit.cpp#L374 Using deprecated casting style. Use static_cast<int>(...) instead (readability/casting) and src/TAccessibleTextEdit.cpp#L525 Tab found; better to use spaces (whitespace/tab)

I added that call thinking that some locales with many wide characters would have a wide ‘a’ character as well. I tested with zh_TW and it wasn't the case, so remove it. Thanks to Stephen Lyons for noticing the issue.

Thanks to ironcross32 for the suggestion.

vadi2 · 2022-04-30T13:07:16Z

See #3342 (comment) for an update. I'll continue this work - will start a new PR as I haven't got the rights to push in here.

vadi2 and others added 3 commits February 21, 2022 22:36

Add in files

d74de2a

Wire it in

b686513

.gitignore: Ignore ‘compile_commands.json’

52bbe6a

bauermann requested review from a team as code owners February 23, 2022 19:47

mudlet-machine-account added this to the 4.16.0 milestone Feb 23, 2022

bauermann mentioned this pull request Feb 23, 2022

Make Mudlet's text window readable by screenreaders #3342

Closed

github-actions bot reviewed Feb 23, 2022

View reviewed changes

SlySven reviewed Feb 24, 2022

View reviewed changes

bauermann added 3 commits February 26, 2022 18:00

TAccessibleConsole: Remove qDebug() statements

a6e4c2f

TConsole: Set accessibleName for mUpperPane and mLowerPane

7be0de2

bauermann added 13 commits February 27, 2022 02:39

Implement caret mode

54621a4

TAccessibleTextEdit::text(): Return contents in the requested range

03f5d66

The current cursor position isn't relevant in the calculation.

TAccessibleTextEdit: Consider offset at the end of the contents to be…

31c0085

… on the last char

TAccessibleTextEdit: Implement method offsetAtPoint()

5f67d35

TAccessibleTextEdit: Implement selection methods

aa2277f

Support only selection 0.

TAccessibleTextEdit: Implement textBeforeOffset(), textAfterOffset() …

fac9557

…and textAtOffset()

TAccessibleTextEdit: Implement the attributes() method

fd9cd85

It's partially based on QAccessibleTextWidget::attributes() from Qt 5.15.

TAccessibleTextEdit: Take scrolling into account in characterRect()

c7d70fd

TAccessibleTextEdit: Implement method scrollToSubstring()

24c2437

TTextEdit: Use more specific ‘TextInsertEvent’ when new lines are added

d4a6361

Also, remove ‘DocumentContentChanged’ event from TConsole::print() methods because both call TTextEdit::showNewLines(), which already sends ‘TextInsertEvent’.

TConsole: Improve accessible description of MainWindow consoles

723dc9f

Thanks to Stephen Lyons for the suggestion.

TConsole: Simplify if/else if statements

b75ba04

Thanks to Stephen Lyons for the suggestion.

bauermann force-pushed the issue-3342-text-window-accessibility branch from 4a8de85 to b75ba04 Compare February 27, 2022 05:43

vadi2 and others added 11 commits February 27, 2022 09:06

Merge branch 'development' into issue-3342-text-window-accessibility

8dbc7c1

TAccessibleTextEdit: Address uninitialized variable warnings from cla…

23cf4a9

…ng-tidy Example warning: warning: variable lengthSoFar is not initialized [cppcoreguidelines-init-variables] int lengthSoFar; ^ = 0

mudlet: Fix clang-tidy warning about slot_caret_mode() not being const.

f7c0510

TAccessibleTextEdit: Fix issues found by CodeFactor

432bd3f

src/TAccessibleTextEdit.cpp#L374 Using deprecated casting style. Use static_cast<int>(...) instead (readability/casting) and src/TAccessibleTextEdit.cpp#L525 Tab found; better to use spaces (whitespace/tab)

Merge branch 'development' into issue-3342-text-window-accessibility

ad74566

TTextEdit: Remove superfluous call to getGraphemeWidth()

a3bfcc4

I added that call thinking that some locales with many wide characters would have a wide ‘a’ character as well. I tested with zh_TW and it wasn't the case, so remove it. Thanks to Stephen Lyons for noticing the issue.

main_window.ui: Change Caret Mode shortcut to ‘Ctrl+Tab’

0564aed

Thanks to ironcross32 for the suggestion.

Merge branch 'development' into issue-3342-text-window-accessibility

944ecd9

Merge branch 'development' into issue-3342-text-window-accessibility

38a6454

Merge branch 'development' into issue-3342-text-window-accessibility

e673121

vadi2 closed this Apr 30, 2022

vadi2 mentioned this pull request Apr 30, 2022

Add: screenreader-friendly main window #6090

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First stab at making Mudlet's text window readable by screenreaders #5985

First stab at making Mudlet's text window readable by screenreaders #5985

bauermann commented Feb 23, 2022 •

edited by Kebap

add-deployment-links bot commented Feb 23, 2022 •

edited

github-actions bot commented Feb 23, 2022 •

edited

github-actions bot left a comment

vadi2 commented Feb 23, 2022

bauermann commented Feb 23, 2022 via email

SlySven left a comment

SlySven Feb 24, 2022

bauermann Feb 27, 2022

SlySven Feb 27, 2022

SlySven Feb 24, 2022

bauermann Feb 27, 2022

Hadi-1624 commented Feb 25, 2022

ironcross32 commented Feb 25, 2022

bauermann commented Feb 26, 2022 via email

bauermann commented Feb 26, 2022 via email

mohammedradwan2003 commented Feb 26, 2022

ironcross32 commented Feb 26, 2022

bauermann commented Feb 26, 2022 via email

bauermann commented Feb 26, 2022 via email

Hadi-1624 commented Feb 26, 2022

bauermann commented Feb 26, 2022 via email

bauermann commented Feb 26, 2022 via email

ironcross32 commented Feb 27, 2022

bauermann commented Feb 27, 2022 via email

bauermann commented Feb 27, 2022

vadi2 commented Feb 27, 2022

vadi2 commented Apr 30, 2022

First stab at making Mudlet's text window readable by screenreaders #5985

First stab at making Mudlet's text window readable by screenreaders #5985

Conversation

bauermann commented Feb 23, 2022 • edited by Kebap

add-deployment-links bot commented Feb 23, 2022 • edited

Test versions

github-actions bot commented Feb 23, 2022 • edited

github-actions bot left a comment

Choose a reason for hiding this comment

vadi2 commented Feb 23, 2022

bauermann commented Feb 23, 2022 via email

SlySven left a comment

Choose a reason for hiding this comment

SlySven Feb 24, 2022

Choose a reason for hiding this comment

bauermann Feb 27, 2022

Choose a reason for hiding this comment

SlySven Feb 27, 2022

Choose a reason for hiding this comment

SlySven Feb 24, 2022

Choose a reason for hiding this comment

bauermann Feb 27, 2022

Choose a reason for hiding this comment

Hadi-1624 commented Feb 25, 2022

ironcross32 commented Feb 25, 2022

bauermann commented Feb 26, 2022 via email

bauermann commented Feb 26, 2022 via email

mohammedradwan2003 commented Feb 26, 2022

ironcross32 commented Feb 26, 2022

bauermann commented Feb 26, 2022 via email

bauermann commented Feb 26, 2022 via email

Hadi-1624 commented Feb 26, 2022

bauermann commented Feb 26, 2022 via email

bauermann commented Feb 26, 2022 via email

ironcross32 commented Feb 27, 2022

bauermann commented Feb 27, 2022 via email

bauermann commented Feb 27, 2022

vadi2 commented Feb 27, 2022

vadi2 commented Apr 30, 2022

bauermann commented Feb 23, 2022 •

edited by Kebap

add-deployment-links bot commented Feb 23, 2022 •

edited

github-actions bot commented Feb 23, 2022 •

edited