Implement Subscribe Events

[LexiconCode]

Greetings most accessibility API framework have their own version of subscribing/monitoring events.

• UI Automation Events and IAccessibleHandler
• atspi-events
• It's not clear for Apple based on nsaccessibilityprotocol if it has an event monitoring in the API.

pywinauto is tooled towards UI automation (UI testing) not as an accessibility (screen readers/voice control) library. These cases are very similar but have a different focus.

• An automation framework automates starting X program and executing predefined X actions.
• Accessibility for instance the screen readers/voice control are interested more in the current in focus control state and/or monitoring specific out of focus elements/controls (example the address URL in a browser)

I will not be able to do this alone but I would like to begin looking at what it would take to implement it within pywinauto. Any thoughts on this would be appreciated.

[vasily-v-ryabov]

Hi @LexiconCode some prototype of script recorder was implemented for "uia" backend in PR #701. We postponed its productization as it is very complicated. The plan looks so:

1. Prepare normal release 0.7.0 from atspi branch which is active for now (we accept PRs to this branch only). No exact ETA, but we have 0.7.0 milestone with planned issues to resolve.
2. Implement/rebase "uia" recorder on top of 0.7.0 release and prepare it as 0.8.0 release. We also have some prototype for "win32" backend which uses DLL injection to monitor Win32 API window messages in the app process directly with transferring data to Python process through named pipe (it's more reliable than sockets).

We also hope to add macOS support soon as normal "ax" backend. Monitoring/recording capabilities on macOS are also possible, but it's a very long term plans probably. It depends on real demand and interest from my students.

[vasily-v-ryabov]

UIA recorder feature is tracked in #332. Also issues with label "New Feature" might be interesting: https://github.com/pywinauto/pywinauto/issues?q=is%3Aopen+is%3Aissue+label%3A%22New+Feature%22

[LexiconCode]

Thank you for the information!

2. Implement/rebase "uia" recorder on top of 0.7.0 release and prepare it as 0.8.0 release.

I made an attempt and it looks very complex to rebase. Due to the age of the PR it's hard to distinguish between its changes without having a deep knowledge of the upstream CodeBase.

[vasily-v-ryabov]

Yes, it's really bloody merge. :) If you really want to learn this code deeply, I can do this rebase in May. The feature itself is also not easy, but code changes look not so scary. Maybe this code can be re-used for nice notification handlers implementation (without fully functional recorder yet).

[LexiconCode]

Yes, it's really bloody merge. :) If you really want to learn this code deeply, I can do this rebase in May.

That would be very generous of you and I would appreciate it. It gives a good place to start.

Maybe this code can be re-used for nice notification handlers implementation (without fully functional recorder yet).

After looking over the code this is what I was going to suggest breaking it out into 2 separate PRs. The 1st focusing on Events to which the recorder could be built off it a later time. So your idea for notification handlers would seem to be a good direction.

FYI: UI Automation is backwards-compatible with IAccessible2 as of the May 2019 Windows 10 update sources

It may be helpful for your students to see the UI out of a framework in another context that in a similar domain. nvda screenreader is an interesting utilization of Microsoft UI automation framework. The code is well commented and highlights some of the challenges particular applications have utilizing accessibility APIs.

My goal is to facilitate the speech recognition community. The SR community is looking for a deeper way to integrate with applications beyond emulating keys/mouse. The purpose being to provide hierarchy of methods to enhance accessibility of software. The goal in pywinauto is the to contribute back rather than re-implementing accessibility APIs ourselves. That way everybody benefits everyone. Some previous work on accessibility from the community.

• OCR and Eye Tracking
• Enhanced text manipulation using accessibility APIs / Implementation
• Mouse Grids

[vasily-v-ryabov]

Luckily rebase for UIA recorder wasn't so hard. This branch contains UIA recorder rebased from atspi: https://github.com/pywinauto/pywinauto/tree/uia_recorder_rebase For Python 2.7 it could start and log something, but for Python 3.7 there are problems with relative imports in recorder_cli.py (we never tested it on Python 3.7+ previously, only Python 3.6 and 2.7). This is entry point script. Example is documented in PR #701 description.