Screen Reader Support

[Neurrone]

Hi,

I've seen some posts dating several years ago on this, am wondering if there's been any updates on this?

Installing the latest version of Mudlet and testing with NVDA, the 2 primary problems I've found are:

• UI elements aren't exposed. E.g, it is unable to read the installer. When attempting to read the main mudlet window, the initial "choose a world to connect" screen works perfectly, but then nothing gets read out. Also couldn't figure out how to access the various dialogs such as triggers, scripts etc from the keyboard.
• Reading incoming output automatically. The way this is being done in Mushclient is via the MushReader plugin that uses the OnPluginScreendraw callback to see the text that is about to be written on screen, and then use screen reader specific APIs to speak that information. If Mudlet supports loading dlls, and has a similar way of getting this information, I can try porting the Mushclient plugin.

[vadi2]

Hi, thanks for your interest!

What you see is the latest that Mudlet has. Mudlet doesn't have the equivalent of that Mushclient callback, but it does use Qt, which has support for accessibility: https://blog.qt.io/blog/2012/06/18/qt-5-accessibility-apis. So in theory, exposing Mudlet to Qt's accessibility engine should do the trick in a cross-platform manner.

If you'd like to give it a go, instructions on building Mudlet are available here! Happy to help with anything if you'd like help.

[chrys87]

Hi,

I've seen some posts dating several years ago on this, am wondering if there's been any updates on this?

Installing the latest version of Mudlet and testing with NVDA, the 2 primary problems I've found are:

* UI elements aren't exposed. E.g, it is unable to read the installer. When attempting to read the main mudlet window, the initial "choose a world to connect" screen works perfectly, but then nothing gets read out. Also couldn't figure out how to access the various dialogs such as triggers, scripts etc from the keyboard.

* Reading incoming output automatically. The way this is being done in Mushclient is via the MushReader plugin that uses the [OnPluginScreendraw](https://www.gammon.com.au/scripts/doc.php?general=plugin_callbacks) callback to see the text that is about to be written on screen, and then use screen reader specific APIs to speak that information. If Mudlet supports loading dlls, and has a similar way of getting this information, I can try porting the Mushclient plugin.

on linux the env:
LINUX_ACCESSIBILITY_ALWAYS_ON=1
needs to be set for QT5 applications to expose the a11y states. Maybe somthing similar is needed for windows. Since i dont dont own an copy of that i could not verify.
i will test this out this evening at least for linux. only thing i see what could make problems is the output window itself (since it might be an custom widget?)
i think good accessibility would be a awsome feature to this MUD client. I would sopport here (since i currently work on Plasma to make it accessible too)

[vadi2]

Yes it's a custom widget for which a11y support isn't implemented, I think that'll be the biggest stumbling block - have a look and let us know.

[chrys87]

woe, that was an fast reply :).
so i assume QaccessibleTextInterface needs to be implemented for this widget to make it work to AccessibleTechnologys.
http://doc.qt.io/qt-5/qaccessibletextinterface.html

But let me check this when i stay at home. I m not blind by myself. But my girlfrind is. so I m interested in fixing a11y issues :). case that you answered that fast i assume you would be intersted in have it fixed too.
maybe we can investigate this? for sure its not a big deal filling the a11y bridge with right informatin.

cheers chrys

[vadi2]

Yeah, that would be the one. It's not something I can pick up right now, but I can certainly help review if you start a PR.

[chrys87]

Ok, i would try to hammer it in. But i assume you can tell me more quick what implements the output window (without need an deeper investigation):
https://github.com/Mudlet/Mudlet/blob/development/src/TCommandLine.cpp
or
https://github.com/Mudlet/Mudlet/blob/development/src/TConsole.cpp
just to have an easy starting point :).

[vadi2]

Yep it's the latter, TConsole.

[celtichawk]

Hey, just jumping in to suppport this. On Linux with Orca it's mostly the same issues. However, the connect to a world.. box isn't displayed properly in LInux, f.ex you can't delete a profile even if you flat review. Hitting tab acts as if the delete buton's not a buton and skips to the text box. Hitting the key to emulate a left click doesn't work either. That being said, I'm just reporting what I've found.
Once connected, the display doesn't auto refresh at all. I have to hit enter to get it to refresh and sometimes even that won't do it. I'm not sure how easy it'd be (I'm no coder) but Chrys knows what he's doing. I'll keep on reporting what I'm finding to narrow things down.o

[Kebap]

Thanks for reminding this issue, I find also very important. In fact, we now have a nice "a11y" label to group similar issues here on github in the future. A few months back was the following attempt to gather and list a few information about problems and solutions already: #1768 Maybe you find interesting read.

[chrys87]

Thanks for reminding this issue, I find also very important. In fact, we now have a nice "a11y" label to group similar issues here on github in the future. A few months back was the following attempt to gather and list a few information about problems and solutions already: #1768 Maybe you find interesting read.

Right, thanks for that information. i was not aware of.
i did an quick investigation of the output window using Accerciser a11y trace debug tool. it doesnt show any ATK lables at all (for output window). so my first assumption was right.

[chrys87]

I wonder if there is a reason for an custom widget instead of using QTextEdit what should provide all needed after an first look. It aso should make caret working ootb as nice side effect.

Is there a reason for? Otherwise i would try to convert this instead of adding a qaccessibleinterface

[vadi2]

Yes, performance and scripting with Lua.

[SlySven]

The trouble you will have is that as well as the text displayed in the TConsole => TBuffer => TTextEdit class hierarchy, Mudlet also has to track on a grapheme-by-grapheme {character-by-character to most people 😉 } basis: the foreground and background colour and some other visual formatting details: bold, italic, underline, struck-out (I also have medium term plans for overline and reverse {foreground/background swap} but you don't need to worry about those just yet).

This means that we have to do some low-level work to paint the text on the screen ((int) TTextEdit::drawGrapheme(QPainter &painter, const QPoint &cursor, const QString &grapheme, int column, TChar &charStyle) const is pretty much the lowest level code for this) - and we do a certain amount of caching of what does get produced so that we avoid redoing the stuff in that method for parts of the screen that has not changed - this combination means that it may be harder for Accessibility functionality to be bolted on and get the information it needs to pass on to differently abled persons... 😞

[vadi2]

I don't think the a11y interface needs to be at the per-character level - it doesn't need to report colours and etc to the user when reading out normal text, does it?

[chrys87]

Yes, performance and scripting with Lua.

hm, i still do not understand why an presentation widget is used to make LUA scripting possible. to me it sounds more like something what should to be handled in deeper levels but not on viewer level. Maybe i can take a look those days to make it more clear to me. Maybe its not so easy to separate the class hierarchy here cleanly somehow.
I do not really understand the performance issues. I mean, this is not an ego shooter but an text game presenter. Just curious about. I somehow cannot imagine that the processing time isn't even notable. Is the default QT widget really that slow?

The trouble you will have is that as well as the text displayed in the TConsole => TBuffer => TTextEdit
class hierarchy, Mudlet also has to track on a grapheme-by-grapheme {character-by-character to
most people wink } basis: the foreground and background colour and some other visual formatting
details: bold, italic, underline, struck-out (I also have medium term plans for overline and reverse
{foreground/background swap} but you don't need to worry about those just yet).

sounds like using XML/ HTML or TeX would be an good idea, this transfers the information already and is unlimited extendable with new format information without need to change the deep structures every time you want to extend it. there are also a lot of fast viewing widgets for that kind of information.

I don't think the a11y interface needs to be at the per-character level - it doesn't need to report colours and etc to the user when reading out normal text, does it?

screen readers has the ability to read the format information. but for sure its also not a need to play most of the games and just an gimmick in first place.

[vadi2]

Yeah, the performance when we started out wasn't good enough - and benchmarking both widgets, then ripping out the existing one and putting the default back in would be more work than plugging a11y into what we've got.

[SlySven]

🤔 I think I had concerns that the VI user would need to be aware of some visual formatting details where the MUD uses that to impart important details (though working out which ones is a separate issue) and the text painting code would be working a grapheme at a time whereas a Screen Reader is probably expecting text in chunks of, say, a sentence at a time. {Actually if it is a sentence at a time we may need to thing a little more about End-of-Line handling so that the reader does not pause at that point - or maybe it will need to do so for multi-line texts so the user can get a feel for what is being presented on screen?}

💭 IIRC Mudlet does not really distinguish internally between LF-CR that it inserts to fit the text to the specified wrapping and those that are included by the sender - we might want to revisit/check how we do this for other reasons (better handling of non-Latinate languages where non-monospaced glyphs get pulled in to show graphemes not in the specified font for a particular TConsole)...

[SlySven]

For an example of where the formatting might be important, DikuMuds and some derivatives mark up the names of enemies in a different colour to warn the user when they appear and perform actions which obviously are likely to be hostile. The main MUD I have played uses asterisks around a red colour applied to the enamy's *name* when it does.

[vadi2]

Sorry, I'm not certain on what are you saying - but we can look at existing clients with screenreader support to figure out how it ought to work.

[SlySven]

I was pondering about:

• A) to what extend we might need to get visual formatting information passed to the screen reader; some of it can be important enough that a visually impaired user still needs to know about it.
• B) whether we will need to pass a screen reader the text in terms of sentences or actual on-screen line-by-line at a time; I guess the former may sound better but the latter will be easier to do however it will also be influenced by our wrapping the incoming MUD text...

[vadi2]

That makes sense, yep. We should also look at how Qt's accessibility APi expects it as...

[devinprater]

For me, most MUD's that show colored text for things also have accessibility settings to prefix the thing with a character, like "you see a #door, a #building, and a #window here," shows that those items are interactable in, say, the Clok MUD. So, for now, I don't think sending formatted text is too much of a priority for now.

[francipvb]

Hello folks,

I am interested inthis. I currently use mushclient, but it is a bit tricky to add, for example, mapper.

It has no built-in gmcp neither, these functionality are implemented in lua, and they're a bit slow.

[lilmike]

Hiya,
I'd also be interested in accessibility for mudlet. Any news? As an aside, anyone know of a way to convert a mudlet map to tintin++? That's the main reason I want to be able to use mudlet -- a mud I play (stickmud) has a mudlet map
-Michael.

[vadi2]

Linux and macOS testing for main window accessibility is ready for testing, have a look at the links in #6090 (comment). Windows is under development and will come in the future.

Please file anything you find as separate issues and thank you!

[vadi2]

Windows is ready for testing now as well, same link.

I'll close this issue as #3342 duplicates this one and we're making progress on this 👍