Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Accessibility for Blind Developers Who Use Screenreaders #14011

Closed
ghost opened this issue Dec 6, 2017 · 117 comments
Closed

Add Accessibility for Blind Developers Who Use Screenreaders #14011

ghost opened this issue Dec 6, 2017 · 117 comments
Labels
feature proposal topic:core usability

Comments

@ghost
Copy link

ghost commented Dec 6, 2017

Operating system or device, Godot version, GPU Model and driver (if graphics related):
Operating System: Arch Linux, might be true for blind users on other operating systems
Godot Version: 2.1.4, may apply to all
Issue description:

The Godot game editor does not work with the Orca screenreader.
I was expecting to at least be able to use Orca's flat review with the editor application, but nothing happened; it is equivellent to turning off the monitor.
Steps to reproduce:
Install the GNOME desktop environment, along with the Orca screenreader. MATE will also work.
With Orca running, open the Godot application. There will be no speech output from Orca.
Link to minimal example project:
@akien-mga
Copy link
Member

This would definitely be a welcome improvement, but I can already say that it will likely take a while to get implemented, if at all. Godot's editor uses its own GUI toolkit, which is the same used in games, and is fully independent from the native toolkits or cross-platform ones such as Qt or GTK+, which would have good screen reader support.

So to get this to work in Godot, it would have to be implemented from scratch in the core Controls. I don't know how hard this is, but probably non trivial.

@vnen
Copy link
Member

vnen commented Dec 6, 2017

I have actually thought about this but I don't even know where to start. Godot editor is very visual, you do pretty much everything with the mouse. We would need a person with visual impairment to give us consulting about this and guide us in this task, advising us with what is actually needed and how it should behave. Without an actual user to provide feedback, it's mostly a game of guessing to implement that.

@godotengine godotengine deleted a comment from drentsoft Dec 6, 2017
@ethindp
Copy link

ethindp commented Mar 11, 2018 •
edited

I would be happy to guide you guys in adding this. I'm blind myself. :) At least, I could refer you guys to frameworks (MSAA and UIA come to mind as possibilities, though UIA is preferable to me at least, since NVDA (https://github.com/nvaccess/nvda) (an open source screen reader) uses it) and beta test it as you guys go. In the meantime, is there an alternative godot editor or method of creating the game by hand, without the editor?

@MarianoGnu
Copy link
Contributor

If this feature is implemented in the core of the engine itself then it would be possible to developeprs to make apps for blind people, not only allow blind developpers to work.
+1 to the proposal

@UrbanFuturistic
Copy link

I'm glad to see this is being looked at and the only thing I have to add about this is that screen reader software is also use by people who have learning disabilities/difficulties such as dyslexia. As much as I hate the way these things sometimes leave people out, I think it could be argued that this broadens the case for such a development.

NV Access (NVDA) would be a good first place to contact as both lead developers are blind and so have a proper insight into what's necessary. Maybe a collaborative effort?

@francipvb
Copy link

Hi,

I think that this will be a huge effort. That can be done but considering native accessibility protocols an implementation made from scratch can be very problematic for users of all platforms.

If you plan to go forward with this, I recommend to you to use the QT model, an abstract accessibility event layer on top of the basic UI controls first and a platform dependent implementation.

Another approach is to use Text To Speech directly. This can be more platform-independent and so much easy to work on. There are some problems with this, too, but they are more related to the engine than the runtime environment.

Hope this can help you deside.

Cheers,

@ethindp
Copy link

ethindp commented May 18, 2018 via email

@ndarilek
Copy link
Contributor

As a partial solution, are there any docs, or even places to look, to implement a game by hand without using the editor? I've seen the command line docs, but I'm wondering if it's even practical/remotely enjoyable to work with the scene graphs and such by hand, thus completely eliminating the need for the editor.

As a more practical example, right now I'm implementing an audio-based Asteroids-like game. Asteroids emit sounds to mark their positions, and the player centers them in the audio field by turning/flying around. I don't particularly care about the visuals, but they'd be useful for debugging purposes so I can ask sighted folks if, say, my mental concept of my ship's orientation matches with the crude cylinders/spheres I'm using for debugging/collision detection. I'm wondering if it's even remotely possible to position a 5-unit-diameter sphere at 0,0, randomly position and move a bunch of 25-unit-diameter spheres, implement wrapping behavior, etc. by hand-coding it all without the need of the Godot editor. Or will it be a mess of editing non-human-friendly XML?

At the moment I am doing this more or less by hand in Rust with an ECS. That's not terrible, but I do wonder if I gain anything by leveraging an actual game engine. Ideally the Asteroids thing is just a proof-of-concept, and I'd eventually like to make richer audio games without having to build all this stuff up by hand. Whether Godot is that solution, I don't know...

Either way, I do think that implementing a full accessibiity interface to the UI toolkit is probably a bit much to ask. If the data files are mostly text but human-unfriendly, a middle-ground solution might be a DSL that exposes a simplified interface for the kinds of tasks I described above (positioning shape primitives, attaching sounds, adding logic to entities. etc.)

@vnen
Copy link
Member

vnen commented May 24, 2018

As a partial solution, are there any docs, or even places to look, to implement a game by hand without using the editor?

I think the easiest way (without hacking around) would be to generate the scenes procedurally via script. You could either do that at runtime or make a script that saves the scenes and resources to disk so you can add to the project. The initial scaffolding would require a project.godot file for the project settings (which can be an empty file made by the OS at first) and a main scene that runs at start (which can also be set and saved via script). Might be complex to manage, but it's possible.

Technically, you can make a script that runs directly from the command line and that script makes the whole Godot project for you (at least I can't think of anything that would prevent that). You can then run and export the game from the command line as well. You would need to open the project in editor at least once for importing the assets (and every time the assets change) but that could be solved by running the headless version of Godot, as some people are doing for use with Continuous Integration servers.

If the data files are mostly text but human-unfriendly, a middle-ground solution might be a DSL that exposes a simplified interface for the kinds of tasks I described above (positioning shape primitives, attaching sounds, adding logic to entities. etc.)

The text resource format is not much hard to read, but still it's probably simpler to just use GDScript to create stuff for the project, since you can use the base engine for writing and reading the format (avoiding compatibility issues).

Unfortunately, GDScript is not the best language for visually impaired people, since the white-space is important. I heard complains from a blind developer about Python because of this.

@ethindp
Copy link

ethindp commented May 25, 2018 via email

@vnen
Copy link
Member

vnen commented May 25, 2018

@ethindp I won't argue about what is or isn't, since I'm not blind myself, but I'll ask you to tone down. People have different backgrounds and might have proper reasons to find it hard to work, while you learned to work with it. The person I know works as a developer for a living, I find hard to believe he "didn't try hard enough" (it's still second-hand for me though, so I won't argue about that).

I've never heard of the headless version of Godot. Perhaps that could
be used to create projects and interact with the engine as well? That
might be a complete solution to our problems at the moment. Where can
I find this version? (Or is it in the Godot package?)

It is called the server platform and works on Linux only. It's not distributed (maybe it'll be once 3.1 is released), so you need to compile yourself to access it. This is just the regular Godot engine, but it has dummy drivers for video in order to not require a graphics card.

I'm not sure how much you can interact with it, likely you can't do it manually. Still, it can be used to run scripts and importing assets, so it's possible to make a project with it if you do everything with scripts (at least in theory).

@ndarilek
Copy link
Contributor

I'll echo what @ethindp says in that indentation shouldn't be an issue. Most modern screen readers have settings to speak indentation automatically (I.e. "8 spaces def hello_world():`, and I know lots of prolyphic blind Python developers.

I wonder if there's even another middle-ground solution--a line-oriented command line tool that can manipulate the scene graph in real-time, drive another instance, and let you save its output. I'm thinking of an interaction style like:

$ godot-shell mygame.godot
Welcome to the Godot shell.

> new entity
Entity created with ID 0.
> add component 0 position
Component "position" added with ID 0.0.
> set component 0.0 [0, 0]
Component 0.0 set.
> save
Scene graph saved.
>

I'm then imagining a separate instance running in another window, displaying whatever the engine is currently outputting, playing sounds, etc. Other commands might spawn a GUI text editor to edit scripts.

Would this kind of workflow make sense if the aim is to create a more simplistic game? I.e. I don't really care what my player/world models look like, beyond them being simple shapes so I can mentally model them, attach sounds, then have a collision detection system that makes some sort of sense. I also think a CLI is a bit easier to prototype and experiment with than is a full-blown DSL.

I may start hacking on this if folks who know more than I do about Godot think it's viable. Been looking for a good CLI interface problem to sink my teeth into, and have a few Rust libraries I'd like to put through their paces. Is the file/data format documented anywhere?

@ethindp
Copy link

ethindp commented May 25, 2018 via email

@ndarilek
Copy link
Contributor

ndarilek commented May 25, 2018 via email

@ethindp
Copy link

ethindp commented May 25, 2018 via email

@ndarilek
Copy link
Contributor

ndarilek commented May 25, 2018 via email

@vnen
Copy link
Member

vnen commented May 25, 2018

The file format is described (at least partially) in the documentation: http://docs.godotengine.org/en/latest/development/file_formats/tscn.html

However, making a completely new tool for this is a lot of work and is bound to become obsolete quite easily. At least you should use GDNative as a bridge (which is a C interface for the C++ engine), that would give access to all information about classes and their properties. As much as I appreciate the effort, I think you first need to understand a bit what Godot has to offer, so you don't reinvent all the wheels.

If you really want to do with Rust, you can try the bindings for GDNative (though I can't attest it's stability): https://github.com/GodotNativeTools/godot-rust

Personally, I would try to do something with the Godot core itself (maybe as a C++ module), since you can then access all the API easily, including all the resource savers and loaders, so you don't need to worry about file formats.

If you're going for a CLI interface, it should be based on arranging the scene tree, which is the most important part of any Godot game. Designing this process beforehand is probably more important: creating an easy way to understand how the tree is laid out and tools to move nodes to a specific tree position should be the cornerstone of the design. If you get that, you don't even need to limit the feature-set, since pretty much anything can be done with the tree.

After that, you need a way to set properties of the individual nodes, including the resources that are used to give then a proper functionality (e.g. a CollisionShape2D node needs a Shape2D resource to give its shape). Scripts are also resources, so they can be set via this interface as well (though you might want to give them a special status). This would substitute what the Inspector is in the GUI.

Also, I would make commands as close to GDScript as possible. For instance, you want to create a Sprite, you could simply do Sprite.new(), which is the same you would do in a script. This way the learning curve for someone already used to scripting would be less steep.

For audio, Godot uses players and buses. There are players for positional audio (both 2D and 3D) and a regular one for non-positional audio, which are all nodes in the tree as well. Surround is also supported (though I never used it). To work with the buses, you need to interact with the AudioServer, to set volume, mute or solo them, and add effects. You might be able to use the AudioBusLayout resource directly, but I'm not sure how much of it is exposed to the external API. The effects are just resources, so they can be edited by the same interface.

All of this in my humble opinion, of course. I have used Godot for quite a long time now, so I might be somewhat biased.

@ndarilek
Copy link
Contributor

ndarilek commented May 25, 2018 via email

@vnen
Copy link
Member

vnen commented May 25, 2018

GDNative is meant to make games, essentially replacing scripts. But as I said before, you can do pretty much anything with scripts, and GDNative uses the same API.

@francipvb
Copy link

Hi,

For scene or alike and complex objects, you can use a format like this file.

In this case, it is just for a map for a project called audioquake, but it can be an example. P.D: Don't try to build this, it may not work.

For GUI, you can also use QT5.11, I saw that it includes many improvements for accessibility. In this case, you will only need to add some sort of accessibility support for hard custom controls like the camera.

Cheers,

@ethindp
Copy link

ethindp commented May 26, 2018 via email

@ndarilek
Copy link
Contributor

ndarilek commented May 26, 2018 via email

@ethindp
Copy link

ethindp commented May 26, 2018 via email

@vnen
Copy link
Member

vnen commented May 26, 2018

I think it would have been more correct for me to ask if I needed the
GUI to create games. I'm happy running them from the GUI, I just want
to develop them on the console. And it looks like I can.

You don't need to GUI to run games. In fact, the editor just creates another Godot process with the arguments to run the game (with all the debugging options as well). Running the game is a non issue: you can simply run Godot from the project's directory without arguments, the default is to run the project.

The editor is pretty much a Godot "game" itself. It uses the available nodes and extends them. The part about importing, it's not that you need the GUI, you just need the importing tools that comes with the editor. Essentially you just put all assets in the project folder (in any structure your like) and when you open the editor it'll import everything.

"Importing" in this case means storing metadata (if you want to loop and trim audio, or apply filter to an image, etc.) and sometimes converting to a format that Godot understands and can use (like importing a model to a Godot mesh). The import process can only be done by the tools build (i.e. the editor), even if the headless version.

When you export the game, only the imported assets are packed, the original ones are ignored. For the game itself this process is transparent: you can treat the assets as if they were in the original location, the loaders will know to get the imported version.

Just like the editor is just a game, you can make your own "game" that is actually a tool to make games. I mean, you can even create a script that extends MainLoop or SceneTree and run directly via terminal with:

$ godot -s my_script.gd

The problem is that you want a true CLI tool, Godot won't make it easy. Even when running a script it'll open a window, and everything sent via stdin will be ignored.

I wonder if we can somehow make the engine not only an
engine with an editor and such, but a library too? Like, your given
the C++ libraries (static or dynamic libs) and can build your game
however you like, using Godot for all the game engine stuff (i.e.
audio and such).

While the editor and the engine are made for each other, you're not entirely dependent of the editor to make the game. You cannot split Godot into pieces though, you need to take the whole package (although you can disable most of the modules without problem). Since the editor is a Godot game, it's technically possible to replace the code by your own game. That would mean creating the scenes on the fly with code.

Godot is not meant to be used as a library or framework, and likely won't ever be, but you can hack around it and make a MainLoop implementation that does not even use nodes at all, instead just call the servers directly. Of course, you would need to build the engine from source, but I imagine this is the least of the concerns.

Any of this is technically possible, and maybe not that hard, but it's venturing into uncharted territory.

@ndarilek
Copy link
Contributor

ndarilek commented May 26, 2018 via email

@ethindp
Copy link

ethindp commented May 26, 2018 via email

@ndarilek
Copy link
Contributor

ndarilek commented May 26, 2018 via email

@ndarilek
Copy link
Contributor

ndarilek commented Sep 20, 2019 via email

@francipvb
Copy link

Oh, sorry, thought you saw this. Don't follow the godot-tts download instructions since they assumed I could get Appveyor working. And the last bit about losing focus on game quit is no longer true. Good luck, hope this works under Windows.

Apparently this link is to a private repo...

Thanks,

@ndarilek
Copy link
Contributor

ndarilek commented Sep 21, 2019 via email

@ndarilek
Copy link
Contributor

ndarilek commented Sep 24, 2019 via email

@ndarilek
Copy link
Contributor

ndarilek commented Sep 24, 2019 via email

@Faless
Copy link
Collaborator

Faless commented Sep 25, 2019

I can't get 3-D audio playing, and put together this example to make things as simple as possible, but it doesn't work. My intent is to create a single 3-D spatial node with an audio stream rumble and listener as children, then play the rumble. This works in 2-D, but not in 3-D. Am I doing something wrong here?

You need to add a 3D Camera to the scene. You can then put the listener as a sub-child of the camera and move the camera, or move the listener directly. That should fix the issue...

which I think is a bug, since no error/warning whatsoever is displayed in the editor when no camera is present and sound is not played.

There are still wrinkles, but working with the Godot editor as a totally blind developer is doable so far.

This is heart-warming to hear, thank you for your work in making Godot more accessible.

@ndarilek
Copy link
Contributor

ndarilek commented Sep 25, 2019 via email

@Faless
Copy link
Collaborator

Faless commented Sep 25, 2019

Ah, I thought the listener was enough

I think it should be, it might be a bug, I don't have much knowledge in that area.

Can I add the camera and prevent it from rendering while still keeping whatever property makes the listener work?

You can put all the 3d stuff in a Viewport and it will not render (remember to enable the listener property).

I do want to be able to render something to 2-D, just not a full 3-D scene. Or can I tweak the 3-D camera to render 2-D somehow--give it an orthogonal perspective maybe?

You can render both 3D in 2D (via ViewportContainer) and 2D in 3D via (Mesh, Material, ViewportTexture).
There is some documentation here:
https://docs.godotengine.org/en/3.1/tutorials/viewports/viewports.html
and demo projects here:
https://github.com/godotengine/godot-demo-projects
under the viewport subfolder.

@ndarilek
Copy link
Contributor

ndarilek commented Sep 25, 2019 via email

@ndarilek ndarilek mentioned this issue Sep 26, 2019
Merged
akien-mga pushed a commit that referenced this issue Sep 27, 2019
@ndarilek
#14011: Implement TreeItem.get_button_tooltip(column, idx).
5629a00 
When added to `TreeItem`, buttons are given tooltips. When returned via `get_button(...)`, however, the button is a `Texture` and the tooltip information isn't included.

For accessibility purposes, it is useful to have access to the tooltip text. As such, we can retrieve a button's tooltip to use as a button label.
@ndarilek
Copy link
Contributor

ndarilek commented Sep 27, 2019 via email

@Faless
Copy link
Collaborator

Faless commented Oct 8, 2019

OK, hitting another odd issue I need help with. If I add a Raycast2D node, then try to set its collision mask property via the editor, the menu of layers pops up when I press Enter on the button. But pressing Enter on a layer doesn't seem to close the menu and select a layer.

Sorry about the late reply.
Yes, the PopMenu is not closed automatically in that case.
You need to press Esc to close it.
The idea, is that when you set the collision layer (which is a bitmask field) you might want to set more than one bit at once, so the popup stays open.
The relevant code is setting PopupMenu.hide_on_checkable_item_selection to false.
See:
https://docs.godotengine.org/en/3.1/classes/class_popupmenu.html#class-popupmenu-property-hide-on-checkable-item-selection

https://github.com/godotengine/godot/blob/master/editor/editor_properties.cpp#L799

https://github.com/godotengine/godot/blob/master/scene/gui/popup_menu.cpp#L1145

@ndarilek
Copy link
Contributor

ndarilek commented Oct 8, 2019 via email

@Faless
Copy link
Collaborator

Faless commented Oct 8, 2019

Next question, is there some way to intercept and filter touchscreen interaction before it reaches Controls or other nodes?

You can use the Control._gui_input(event) in the root GUI element or even Node._input(event) (possibly even in the root viewport).
You can then use SceneTree.set_input_as_handled() or Viewport.set_input_as_handled() to block them after checking the input type for e.g. event is InputEventScreenTouch.

Check out:
Input event flow:
https://docs.godotengine.org/en/3.1/tutorials/inputs/inputevent.html

_input method in Node.
https://docs.godotengine.org/en/3.1/classes/class_node.html#class-node-method-input

set_input_as_handled() method in SceneTree.
https://docs.godotengine.org/en/3.1/classes/class_scenetree.html#class-scenetree-method-set-input-as-handled

_gui_input in Control
https://docs.godotengine.org/en/3.1/classes/class_control.html#class-control-method-gui-input

@ndarilek
Copy link
Contributor

ndarilek commented Oct 9, 2019 via email

@ndarilek
Copy link
Contributor

ndarilek commented Oct 11, 2019 via email

@ellenhp
Copy link
Contributor

ellenhp commented Oct 12, 2019

@ndarilek,

Would substituting an is_class check for the get_class check solve this problem? The docs for that method are here.

@ndarilek
Copy link
Contributor

ndarilek commented Oct 14, 2019 via email

@ellenhp
Copy link
Contributor

ellenhp commented Oct 14, 2019 via email

pchasco pushed a commit to pchasco/godot that referenced this issue Oct 23, 2019
@ndarilek @pchasco
godotengine#14011: Implement TreeItem.get_button_tooltip(column, idx).
899034a 
When added to `TreeItem`, buttons are given tooltips. When returned via `get_button(...)`, however, the button is a `Texture` and the tooltip information isn't included.

For accessibility purposes, it is useful to have access to the tooltip text. As such, we can retrieve a button's tooltip to use as a button label.
@KoBeWi KoBeWi mentioned this issue Nov 9, 2019
Closed
@ndarilek
Copy link
Contributor

ndarilek commented Jan 3, 2020

We may be able to close this one soon, OTOH I've had lots of questions answered here that IRC/the forums haven't helped with. That's true again. :) Hopefully I've kept the S/N ratio high, but when I need help with something, it's generally above and beyond what I can find in a tutorial or book, and there are plenty of skilled folks watching this issue and helping me build this support out.

DId lots of work on this over the holidays. My godot-tts plugin now supports Android and HTML 5, and I've exported accessible games to both platforms. I also did some initial work on touchscreen accessibility. On my Linux desktop, UIs can now be explored accessibly on my $70 HDMI touchscreen. Simple touches speak the UI control being touched, a double-tap anywhere on-screen activates the last focused control, and a quick swipe right/left acts like Tab/Shift-tab and moves focus between elements.

Unfortunately, swiping doesn't work at all on Android, and I'd appreciate help figuring out why. More specifically, the swipes themselves are detected just fine. They then inject ui_focus_next or ui_focus_prev actions using code like this:

func press_and_release(action):
    var event = InputEventAction.new()
    event.action = action
    event.pressed = true
    get_tree().input_event(event)
    event.pressed = false
    get_tree().input_event(event)

func swipe_right():
    press_and_release("ui_focus_next")

func swipe_left():
    press_and_release("ui_focus_prev")

And those actions don't trigger on Android, even though the swipe_right/swipe_left functions run just fine. Any idea why that might be?

I plugged a keyboard into my phone, and Tab at least triggers ui_focus_next. So the action works in terms of being recognized, but as generated via the above code it doesn't seem to trigger. I also tried Input.action_press and Input.action_release, but that made things stop working even on the desktop. So clearly my code does something Input.action_press doesn't, but it doesn't do enough to make Android happy. I tried to find where the events were converted to actions, but it isn't clear to me whether or not these paths are platform-specific. Clearly they have to be since Linux/X11 and Android diverge.

Running out of ideas here and am open to suggestions. Thanks for the help so far.

@vnen vnen mentioned this issue May 31, 2020
Open
@akien-mga
Copy link
Member

akien-mga commented Jun 2, 2020 •
edited

Superseded by godotengine/godot-proposals#983.

Note that this might be worth splitting in a separate proposal again. We're closing old proposals on this repository to encourage migrating them to the new godot-proposals tracker where we want all proposals to reside and be discussed.

@bruvzg bruvzg mentioned this issue Dec 23, 2021
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature proposal topic:core usability
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests