Make Mudlet's text window readable by screenreaders

[vadi2]

Brief summary of issue / Description of requested feature:

Currently when you use a screenreader, it is able to read menus and dialogs of Mudlet - but not the actual window where the game text is displayed.

This is because menus and dialogs in Mudlet are standard Qt widgets which already have support for accessibility, whereas the game text widget is a handcrafted and very fast widget for rendering text - which does not have a11y support yet.

This issue is about adding support to it: the TConsole / TTextEdit classes.

Steps to reproduce the issue / Reasons for adding feature:

1. Better accessibility for screenreaders

Error output / Expected result of feature

Expected result is that NVDA on Windows, built-in macOS reader, KDE and Gnome's accessibility are able to read text as it comes from the game and the widget is navigatable (as in, to go back to read text) in a standard way - as an impaired player would expect.

Thus this shall be implemented using Qt's accessibility framework as that will automatically handle the OS-specific details for us: https://doc.qt.io/qt-5/accessible-qwidget.html

Extra information, such as Mudlet version, operating system and ideas for how to solve / implement:

Mudlet 4.4.0

Bountysource

This issue will be considered closed when at least 2 visually impaired users sign off on usability.

We're new to developer bounties and this is our first foray into it - so we expect a few bumps along the road :)

[svipal]

Beginning to tackle this. Expect a WIP /PoC next week !

[svipal]

Already said it on discord, but will say it here : Will take a looot longer than that ! :P

[vadi2]

@mpconley has donated some work already - see if https://github.com/Mudlet/Mudlet/compare/add-mpconleys-accessibility-work helps you out.

[svipal]

Hey, sorry for all this , but work has piled up elsewhere and I've gone nowhere - I feel like it's unfair for me to claim I'm working on this and potentially reserve a bounty when I'm not. I'm abandoning this for now0, will come back in a month or so when things calm down.
Update : Will not come back

[ghost]

Hate to look snide, but someone has to point the elephant in the room: you can do whatever you want with the textbox, but for a blind person, this app's value will still be outweighted by the entirety of its UI being a nightmare to use with keyboard alone. No, really, take a few minutes to memorize the relevant forms, close your eyes then try to create a script without touching the mouse. At least on Linux, you'll find that you can't even navigate them: the toolbar buttons you're so fond of don't receive focus, and there are text controls that capture the focus with no way to leave them. With scripts being unusable, the app isn't any more useful than, say, a telnet client. And you can't easily fix the usability - there's more than enough technical debt in the app to stall you for months if you try to. All in all, you should probably play to your strengths and ignore a11y altogether, lest you waste too much time on it and start bleeding the core user base.

[vadi2]

We're aware it's a lot of work, but internalization was a lot of work and we've achieved it - and brought Mudlet to more people out there. We, people making the client, will achieve accessibility too.

[ghost]

Okay, I'll see to finishing mpconley's work then. Will talk to you later.

[ethindp]

I'll see what I can do for the output window. I'm not sure how far I'll get; I have college work to do too, but I'll give it a go.

[SlySven]

🤔 I had wondered whether it would be possible to convert the TTextEdit class to render it's output in the form of a QTextDocument rather than painting out each glyph at a time with QPainter::drawText(...) onto a QWidget - that would give us automatic access to the full Qt-Screen reader interface - but at the cost of having to redo our selection and manipulation tools. We (and our Lua subsytem) uses a line + column indexing system whereas the QTextDocument class is indexed by a single integer offset (in UTF-16 / QChars) from the start. There is the QCursor class which sort of works in the middle which might help with the translation...

[ethindp]

Yeah, you'd need to redo a lot of that, but it would be worth it. The question is, what would it break?

[vadi2]

And how would the performance be like as well? This is the main reason we are using a custom widget to begin with.

[SlySven]

Strange as it may seem I think the rendering will be faster with a QTextDocument - as it will use Qt's own (probably optimised) layout code. The difficulty is that performance is NOT (IMHO) the reason we are using a custom set of widgets - but instead it is because we want/need track the font formatting (fore/background colour, bold, italic, under/over/through-line and colour reversed) aspects for each grapheme and the only way we have to do that is via a separate TChar alongside each QChar in the text.

It ought to be possible to set/get all those aspects (apart, perhaps from the reverse one) by manipulation of the "characters" in a QTextDocument but it is not obvious how efficiently/fast that could be done in comparison with the current system which acts on the (std::deque<std::deque<TChar>>) TBuffer::buffer directly...

[vadi2]

Hey @ethindp, how're you coming along? Need any pointers?

[ethindp]

I sadly haven't been able to work on this lately, so I've made little progress. I'm not sure how far I'll get today or tomorrow since I've got college tomorrow and I'm getting my COvID vaccine today. We'll see how things go.

…

On 4/5/21, Vadim Peretokin ***@***.***> wrote: Hey @ethindp, how're you coming along? Need any pointers? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #3342 (comment)

-- Signed, Ethin D. Probst

[joecrayne]

I would like to work on this to collect the bounty. I've already looked at the code and have some ideas on design. I would avoid the accessibility framework because I can give you a better user experience by providing direct support for text-to-speech (via QT api).

[ethindp]

You will have to design all of the accessibility functions yourself if you do it this way. A screen reader contains various functions that many people who use this will expect. So if you use the native QT text-to-speech API, you're going to need to provide all of these yourself. That includes things like interrupting and pausing speech, for example.

On Sat, Jul 3, 2021, 11:06 joecrayne ***@***.***> wrote: I would like to work on this to collect the bounty. I've already looked

at the code and have some ideas on design. I would avoid the accessibility framework because I can give you a better user experience by providing direct support for text-to-speech (via QT api).

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[joecrayne]

I have it speaking text as it appears on the screen. If you press any key with the input box focused it immediately stops the text playback. The accessibility API is not going to give the right user experience for the mud sessions. First and foremost this should be playable. My thinking is that I should be able to sit with my hands on the keyboard and wear a blindfold and play the mud by listening instead of looking. A generic play/pause feature would just be an awkward encumbrance. Also, the accessibility api will not allow you to configure different voices for different hosts which is really the correct thing to do considering Mudlet allows you to connect to multiple hosts with different nationalities and languages simultaneously. TTS voices are specialized to specific locales.

So the UI I have in mind:

1. text is automatically spoken as it arrives from the server.

2. a keypress stops playback.

3. only the focused mud session tab is spoken

4. Perhaps there should be a way to restart playback for all text that has arrived since the last send. Ctrl-R or something.

5. Each host can have a different voice configured. Switching tabs will switch voices.

I've already started but do I have the go ahead to implement this? I've never done a software bounty and want some kind of assurance that I'm following protocol and that I'll be able to collect.

[ethindp]

How familiar are you with screen reading technology? Because wearing a blindfold is not going to provide you the same experience as a blind individual who uses a screen reader everyday, all the time, to use their computer, their phone, their watch, their television, etc. A screen reader provides various functionality that if you were to utilize their APIs, you would get for free. This isn't just the interrupting of speech or speech synthesis. This includes more useful functionality like custom dictionaries, custom input gestures for keyboards, mice, touch screens, and braille devices, braille output and input, custom scripts and plugins for particular applications, and so on. A screen reader is also going to provide a consistent user experience across practically all applications, regardless of the application in question, so long as the application implements the accessibility frameworks. On Windows, you have UI automation, I accessible too, MSAA, and so on. On Linux you have ATK, and ATK is generally available across the BSD operating systems as well. On Mac OS you have the accessibility framework. The screen readers also give you access to custom synthesizers that are not provided by the operating system, and there are people who use these custom synthesizers and who prefer them over operating system provided speech synthesis. I understand, I think, why you wish to go for a custom implementation, but if you are going to do that, I strongly recommend that you also provide screen reader support as well. You do not need to implement the accessibility frameworks if that will be to troublesome or too difficult. There are certain applications that do implement custom accessibility handling because they do things differently and trying to constrain what those applications do into the constraints of accessibility frameworks is simply too difficult or to costly of a task. However, generally applications that do provide custom accessibility handling and integration with native speech synthesis systems on the operating system, also provide screen reader text-to-speech output so that that is one less thing that a blind user needs to customize and configure, and that application can also get things like custom dictionaries for free, without having to implement anything, because the screen reader provides all of that for you. Custom dictionaries, for instance, are especially useful, particularly for people who do not know English well or speech synthesizers that pronounce two words, one spelled wrong and the other spelled right, identically, or even speech synthesizers that pronounce words that are spelled correctly wrong. I would be happy to provide more feedback, as well as linking you to a forum, a very large community actually, of blind individuals who would be happy to help you out with this project. I was going to attempt it myself but I got majorly distracted with other things and I don't really have the time to do it now, unfortunately. But if there is any way I can help, I would be happy to do so.

On Sat, Jul 3, 2021, 17:59 joecrayne ***@***.***> wrote: I have it speaking text as it appears on the screen. If you press any key

with the input box focused it immediately stops the text playback. The accessibility API is not going to give the right user experience for the mud sessions. First and foremost this should be playable. My thinking is that I should be able to sit with my hands on the keyboard and wear a blindfold and play the mud by listening instead of looking. A generic play/pause feature would just be an awkward encumbrance. Also, the accessibility api will not allow you to configure different voices for different hosts which is really the correct thing to do considering Mudlet allows you to connect to multiple hosts with different nationalities and languages simultaneously. TTS voices are specialized to specific locales.

So the UI I have in mind: text is automatically spoken as it arrives from the server. a keypress stops playback. only the focused mud session tab is spoken Perhaps there should be a way to restart playback for all text that has

arrived since the last send. Ctrl-R or something.

Each host can have a different voice configured. Switching tabs will

switch voices.

I've already started but do I have the go ahead to implement this? I've

never done a software bounty and want some kind of assurance that I'm following protocol and that I'll be able to collect.

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[vadi2]

I agree, it's vital to use the accessibility framework for the bounty. For something that doesn't use the framework and relies on Mudlet's/Qt's built-in TTS features, check out a package made by @atari2600tim: https://forums.mudlet.org/viewtopic.php?f=6&t=23015.

[joecrayne]

I am seeing-impaired and I do have the computer read books to me for entertainment. Mudlet is similar and I think my work would have made a lot of people happy, but if you don't want it, you don't want it. The screen reader tech already works for me in Mudlet about as well as it works anywhere on my system. Have you done testing lately? QT automatically plays with it (even your custom widgets). Go ahead and link a community, but I'm doubtful there's a real mud player anywhere who would prefer the one-size-fits-all screen reader to a directly designed interface for ergonomic play.

[vadi2]

OK, could you put it up somewhere for us to test with?

[keneanung]

Hold now! Please don't start arguing who does the right thing or not. I'm sure there is no one size fits all answer in this question, so let's not waste energy there.

I'd suggest we wait and see what the experience is in the end. There might be multiple ways to the same goal. In the end, we as core developers will need the assessment of the visually impaired community whether the result is useful.

So I'd like to point out that a PR with some kind of Screenreader Support does not automatically get the bounty. If you want to avoid moving into the completely wrong direction, intermediate versions for testing are very useful.

[atari2600tim]

I agree, it's vital to use the accessibility framework for the bounty. For something that doesn't use the framework and relies on Mudlet's/Qt's built-in TTS features, check out a package made by @atari2600tim: https://forums.mudlet.org/viewtopic.php?f=6&t=23015.

My script is very basic and pretty much is a command line front-end for the existing TTS features paired with a one-liner trigger that speaks all incoming text. I didn't make any effort for things like navigating through the backlog, it only will read the most recent few lines. It also says extra things like "Stopping" when you stop.

If someone wanted to, with zero lua programming and just using my script together with key binding interface, one could make keyboard shortcuts that just run the various aliases I wrote. That would accomplish something that rereads recent lines and adjusts voice settings and whatnot with keyboard shortcuts.

With a minor amount of lua programming to upgrade my script, one could navigate through the backlog forward and back to read whatever piece you want. Event engine lets a script run different code between each queued string, or when switching focus between profiles. Switching tones or voices or speed of voice based on context can be done.

Long story short, my script shows the basics of what can be done with the existing features, but it is by no means utilizing everything, and not at all trying to be something enjoyable to use. I did maybe a quarter of what is in this thread. So one shouldn't look at my script and conclude that this is everything existing Mudlet can do. The program already does more than enough to enable blindfolded play.

But blindfolded play is not what people are aiming for. Users are going to want something that interacts with their own software properly instead of a substitute simulating it. A simulation can get pretty far and some will be content with just that much. Personally it fits my needs because I was curious to hear how wordy my game is, but I'm also not relying on this to actually play and I don't have muscle memory where I would expect to interact with it in a consistent way similar to other software.

[joecrayne]

Users are going to want something that interacts with their own software properly instead of a substitute simulating it.

The accessibility API, regarding the screen reader, on linux is just a layer over the more flexible speech synthesis API which is what we are talking about using. The accessibility layer only removes flexibility such as choosing the voice and locale. It's a higher level, less flexible, interface. All the software that interacts as a screen reader backend actually does so by being a speech daemon back-end. I challenge you to find any software anywhere that interoperates with the screen reader system on linux but that does not interoperate with the speech synthesis daemon for which the QT tts api is an interface to. You won't find it. Your objections are purely theoretical and wrong.

[ethindp]

@joecrayne This is incorrect. ATK is not a wrapper around Speech Dispatcher at all. ATK provides a framework that user interface toolkits and screen readers can hook into to provide an accessible and consistent user interface. On Linux and BSD (and possibly MacOS) there is, currently, only one speech synthesis daemon, however that does in no way mean another won't come along. On Windows there are at least three: SAPI 4, SAPI 5, and One Core.
This also discards the possibility of screen readers interfacing with custom speech engines, as NVDA and JAWS do. On JAWS speech synthesis drivers can utilize custom speech synthesis technologies that are not provided by the operating system or that are unavailable under SAPI or One Core. (A screen reader is also far more complicated than what you are trying to do here.)
Again, I would strongly encourage you to do one of the following:

1. Interface with the native accessibility frameworks that are provided by the operating system unless such an interfacing would be too costly to work with, as this will provide a consistent user experience because Mudlet is based on QT already, and therefore interfacing with the native accessibility frameworks provided by the operating system will unify the user experience across the entire application; or
2. Continue with your custom accessibility implementation but provide screen reader output via a library like Tolk on Windows, which also provides braille output, something that some blind users will wish to have because they use braille displays alongside their computers.

Again: a blindfold or listening to audiobooks does not provide the real blindness experience. It is a mockery of the real thing. I have lived with blindness my entire life, and this forum is a large community full of blind people. They will all tell you something like what I and others are telling you here. I would strongly encourage to listen to people who have actually used the technologies you are simulating instead of assuming. Unless, of course, you use a screen reader for everything you do regarding technology yourself.

[joecrayne]

On Linux and BSD (and possibly MacOS) there is, currently, only one speech synthesis daemon, however that does in no way mean another won't come along. On Windows there are at least three: SAPI 4, SAPI 5, and One Core.

My code currently seamlessly integrates two very different speech synthesis systems: espeak and festival. The "one" speech daemon on linux is actually itself a plugin architecture designed to accommodate various unrelated speech synthesis systems. My Mudlet has a speech tab that lets you select from available voices and you don't even have to know whether they are provided by espeak or festival.

The screen-reader UI is such that you have to highlight text with the mouse. This is not a very good UI for speech enabled mudding. I'm not sure how useful it is to a braille device user, but I'm willing to add a braille device support criterion if there is a specific braille device simulator for linux that you can recommend for development.

As I already pointed out, the accessibility API is already being used by QT without exception. If that is your real requirement for the bounty, then congrats, award it to yourselves because despite what the description of this ticket says, there's no widget in Mudlet that does not already support the screen reader in my tests. I can hilight text in both the main console window and in the edit box used for input, and the screen reader will read it. It occasionally gets into a bad state and misses things, but that is not Mudlet's fault. It's global behavior of the screen reader on my system (Linux) and it's another reason to avoid relying solely on ATK for catering to your sight-impaired users.

[vadi2]

@bauermann hey, just curious how are you going with this? Is anything being a blocker?

[ethindp]

I would be happy to test this on my machine -- I'm a full-time Linux user so would be happy to test the Linux side of things.

[bauermann]

Hello Vadim, Em segunda-feira, 10 de janeiro de 2022, às 03:35:05 -03, Vadim Peretokin escreveu:

@bauermann hey, just curious how are you going with this? Is anything being a blocker?

Sorry for being silent for so long. There is no blocker, thanks for asking. I implemented everything and rebased the branch on top of a newer upstream version. I also reorganized the commits. I was in the middle of testing things when the holidays happened and also a few other tasks got in the way. I’m planning to submit a pull request by the end of January. I just updated my branch at https://github.com/bauermann/Mudlet/tree/issue-3342-text-window-accessibility if anyone wants to see where I’m currently at. Cheers.

[ironcross32]

Yeah I've maintained a certain amount of curiosity about this myself.

[bauermann]

Hello Ethin, Em segunda-feira, 10 de janeiro de 2022, às 13:25:51 -03, Ethin Probst escreveu:

I would be happy to test this on my machine -- I'm a full-time Linux user so would be happy to test the Linux side of things.

Thank you for the offer! That will be very helpful. I’m hoping to have a merge request ready for testing in about two weeks.

[vadi2]

That's great! Looking forward to it.

[bauermann]

Em segunda-feira, 10 de janeiro de 2022, às 15:40:43 -03, ironcross32 escreveu:

Yeah I've maintained a certain amount of curiosity about this myself.

Thank you for your interest!

[Hadi-1624]

I'm looking forward to test this on windows and provide feedback

[vadi2]

I've had a peek at the branch - @bauermann if you could, please merge latest branch in as it fixes a crashing issue we had at that point in time of development.

[bauermann]

Em terça-feira, 11 de janeiro de 2022, às 10:14:20 -03, Hadi Rezaei escreveu:

I'm looking forward to test this on windows and provide feedback

That’s great! It will be very helpful, thank you.

[bauermann]

Em quinta-feira, 13 de janeiro de 2022, às 07:57:58 -03, Vadim Peretokin escreveu:

I've had a peek at the branch

Thank you!

- @bauermann if you could, please merge latest branch in as it fixes a crashing issue we had at that point in time of development.

Ok, will do. I’ve been meaning to ask: what is your preference regarding merging vs rebasing/rewriting history? I’ve mostly worked on projects with patch-based workflows, thus I’m more used to rebasing my development branch over time and crafting each commit as a “semantic unit” until the commits are merged upstream. Is it ok for me to rebase my branch on top of the current development branch, or is it better if I simply merge it?

[vadi2]

Do whichever you're most comfortable with - in the end when we merge it gets squashed, so we can look at the commit history and clearly know when something was in or no.

[bauermann]

Em quinta-feira, 13 de janeiro de 2022, às 13:13:16 -03, Vadim Peretokin escreveu:

Do whichever you're most comfortable with - in the end when we merge it gets squashed, so we can look at the commit history and clearly know when something was in or no.

Ok, thank you. I’ll keep doing the rebases and history rewriting then. Old habits die hard. :)

[mohammedradwan2003]

hey,
any news about the accessibility draft
i can test it on pc
i heard about things which i wished to comment on, but i decided not to comment until the version comes out, however some interesting ideas were said, i actually used linux for 2 weeks
it is not a system for life but it is a great system
anyway, until later stay safe.

[bauermann]

Hello,

I finally sent PR #5985 with what I have so far. It’s not intended to be merged, and still has one important issue which I describe in the PR. The PR was made so that snapshot builds can be generated by the CI.

But I think there’s enough there to do some testing. I would appreciate the feedback from everybody here.

[bauermann]

Hello mohammedradwan2003, I’m sorry to take this long to respond. Em sábado, 12 de fevereiro de 2022, às 14:21:58 -03, mohammedradwan2003 escreveu:

hey, any news about the accessibility draft i can test it on pc

Thank you! It will be very helpful if you can test. I did send an experimental version for testing a few days ago, but I don’t think you should spend your time testing it just yet, because from the feedback I received from other people trying it out, it’s not working at all. :/

i heard about things which i wished to comment on, but i decided not to comment until the version comes out, however some interesting ideas were said, i actually used linux for 2 weeks it is not a system for life but it is a great system anyway, until later stay safe.

I would be very interested to hear your comments and ideas!

[bauermann]

Hello,

There’s some news from my part that will unfortunately affect my (admittedly
already slow) pace of development of this feature: I have recently started a
regular job. This means two things:

1. I no longer have a financial interest in completing this feature. Therefore
   if someone comes along and is interested in taking what I currently have,
   complete the job and claim the bounty they should feel welcome to do so.

2. I am still interested in seeing this feature implemented. So in the absence
   of anyone else showing up I will keep working on it. But clearly I will have
   less time to dedicate to it. If I am the one who finishes the implementation,
   then the project can reallocate the bounty money for some other purpose.

[vadi2]

Enjoy the new gig, and big thanks for the support so far!

[bauermann]

Thank you! And thank you for your understanding and your help with this feature.

[vadi2]

Linux and macOS testing for main window accessibility is ready for testing, have a look at the links in #6090 (comment). Windows is under development and will come in the future.

Please file anything you find as separate issues and thank you!

[vadi2]

Windows is ready for testing now as well, same link.

[Hadi-1624]

Thanks to @vadi2 his extensive research and work, I believe this is now completely done

1. Screen readers on Windows, MacOS and Linux will automatically receive the output window and read them as it comes through.
2. IT is now possible for screen reader users to review the output manually if they wish, as Mudlet allows keyboard navigation of the output window, so reading and navigating letter by letter, word by word, line by line and selecting text is possible
3. Special options are added to hide blank lines for faster screen reader reading or replace them with space for screen readers to detect blank lines
4. special option is added to change the hotkey to change the keyboard focus from input to output and back.

I am visually impaired and I use NVDA on windows 10 and can read the output fine with all the work that's mentioned being done to the client.
PS All those features are in a testing version at the moment.

[tspivey]

I haven't tested extensively, but this works on Windows, testing with PR 6187-f91e26dc. I can navigate the output window just fine, and NVDA reads incoming text.

[ironcross32]

Everything within the scope of this issue seems to be complete and working well.