New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable finding of undisplayed menu items #827
base: 0_7_x
Are you sure you want to change the base?
Conversation
I found with our UIA app that: config_item = menu.item_by_path("File -> Configuration", False) did not work. Pywinauto crashes in fact. Traced to the fact that menu.items() does not see the File menu's items until the File menu is clicked and they are displayed. That is the UIA controls don't exist until the they are displayed. This line of code works with this patch, and I can find and then click the configuration item on the FIle menu using: config_item.click_input() and all is good.
104c692
to
35da838
Compare
Codecov Report
@@ Coverage Diff @@
## master #827 +/- ##
=========================================
- Coverage 89.53% 89.5% -0.04%
=========================================
Files 49 49
Lines 20638 20648 +10
=========================================
+ Hits 18479 18481 +2
- Misses 2159 2167 +8 |
Well, well, first Codacy moans about some white space at end of lines - FIXED and ammended commit. Now AppVeyor finds a few failed tests. But I click Details and I see nothing even remotely lucid explaining what failed. Biggest fail for me is AppVeyor ;-). You think there'd be a nice way to see exactly which tests failed and, moreover, if they failed pre-PR or are only failing with the PR merged in? Lack of clarity in the extreme. And codecov, well, of course I added a few lines to a function which, if not tested (covered) is now a weeny bit bigger and hence code coverage would drop a margin (0.04%). I guess what it's saying is that function should have a test! |
It strikes me BTW, that perhaps given I filed #706 earlier this year, that you might not always want menu flutter in order to get the children, and there may be contexts in which they are available with out it. The latter is handled fine in the patch (menu only clicked if no children found), but it still makes the assumption that the MenuItem concerned is opening a submenu not a window. To wit a more general solution would add an optional argument to items(), a flag, which defaults off, and permits menu clicking to search for children. So:
Am loathe to push a change into this PR without some feedback as to pulling it from the team. Waste of my time to keep tweaking this in unpopular directions. I do know, it's an enabler for the app I'm working with, and permits me to navigate menus easily. |
I'm not by my computer, so just a short comment. The failing test_menu_by_mixed_match is a new failure, try to see why it stopped passing. The test_scroll and testCheckBoxes are known as failing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Before looking into new failing test I would recommend thinking on the suggested approach first. Yes, test_scroll
and testCheckBoxes
are known flaky tests. Please ignore them. They have been fixed in atspi
branch which is not merged into master yet. Hope to merge it till end of year.
# element_info. So once we see them bv clicking the menu we cache them | ||
# for subsequent use. | ||
if not items: | ||
if self._cache_items.get(self.element_info.automation_id, None): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wouldn't rely on automation_id
property as a key of dictionary because it's possible to have child items without such property. I can't provide an example right now because of being too busy, but it's worth trying different apps with this approach to test everything works the same way or better.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wow, time flies. I too am only looking at this again now. If automation_id
is not reliable we could just check that it is available and if it is, then cache the item.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Never say "Never!". :) Yeah, that may be better, but why not use self.element_info.name
? Maybe it's a specific menu implementation. Is it Qt5 app?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nope a VCL (Borland) app. I have of course since discovered how neat pywinauto's implementation of metaclasses is such that I simply define a new class derived from this one, and override items, and that is respected and used by pywinauto (the metaclass updates a dict mapping of controls to wrappers). Which means I'm actually free to experiment willy-nilly on broader impacts. Part of me thinks the earlier solution I suggest of an optional argument click_to_find_children=False is impact neutral on existing functionality and does nothing special.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the details! I will think what we can do.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Turns out rather that automation_id
which indeed proved unreliably available, the only key I could find for a caching dict is element_info.element
which seems to be a unique pointer (which Python permits as a dict key). Problem is nothing else I could find anywhere under element_info
provides a unique handle on the element (include name
which is blank on most of our controls).
if self._cache_items.get(self.element_info.automation_id, None): | ||
items = self._cache_items[self.element_info.automation_id].copy() | ||
else: | ||
self.click_input() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
click_input()
is an invasive technique which can make dump_tree()
a.k.a. print_control_identifiers()
blinking with the menu. Item texts are taken into account by best_match
algorithm so dump_tree()
can call it to obtain all possible best_match
names. As I remember, I already fixed similar problem for combo box wrapper pretty recently (I mean last year :) ).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what you mean by "invasive" or what alternative there is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Invasive" means that method .dump_tree()
assumed as read-only. Think about it as a C++ const
method. It calls .items()
for every complicated element, but your implementation of .items()
changes the state of the element (not a const
method in terms of C++) and it even keeps submenu open. More right way to do it is expanding the submenu explicitly in your code, call what you need (list subitems) and then collapse the menu back if you don't need it any more.
Yes, some apps have better design of menu structure so the items are invisible but accessible. In your case a dynamic action is required to list the items.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As per original note above: Part of me thinks the earlier solution I suggest of an optional argument click_to_find_children=False is impact neutral on existing functionality and does nothing special. But I'm loving the easy wrapper override facility that makes experimenting with solutions easy.
BTW, the commit was made with different e-mail not matched with your GitHub account. I always vote for correct account matching commits to count all the contributions properly. And thanks for the attempt! Can we know which kind of application is used in your case? Is it publicly available? |
Sorry for the slow response here. Been nose down, and overlooked this. Not sure how and why the e-mail mismatch happens I have only one github account. Would need more details (clues) to approximate understanding what that is about. As to the application it's a VCL front end: http://docwiki.embarcadero.com/RADStudio/XE6/en/VCL_Overview it dates back to the '90s and is still a going concern, but that history and the VCL front end pose some challenges to automating UI interactions. I'm rolling along though. The main obstacle is that most of the UIA elements don't have names such, and so I am having to find them using class_name and inference, and using |
I found with our UIA app that:
config_item = menu.item_by_path("File -> Configuration", False)
did not work. Pywinauto crashes in fact. Traced to the fact that menu.items() does not see the File menu's items until the File menu is clicked and they are displayed. That is the UIA controls don't exist until the they are displayed.
This line of code works with this patch, and I can find and then click the Configuration item on the File menu using:
config_item.click_input()
and all is good.
There is a strange behaviour I don't understand that the cacheing works around. Essentially in
MenuWrapper.item_by_path()
is called once and clicking the menu works, that is inMenuItemWrapper.items()
,self.children()
contains the menu items after it's clicked.MenuItemWrapper.items()
is called a second time inMenuWrapper.item_by_path()
though, via:menu = next_level_menu(menu, menu_items[i], items_cnt == i + 1)
which calls
self._sub_item_by_text
which again fetches the menu items withitems = menu.items()
.On this second call I find that the
self.children()
inMenuItemWrapper.items()
still has all the children, but mysteriously they have lost all their element_info, they all have no names or texts or classes, are just emptyMenuItems
. Bizarre. Either way, I found making a static cache copy ofself.items()
on the first pass fixes that problem. The Menu is visible of course on both calls, the UIA controls exists. But somehow pywinauto loses all the element info between these two calls toMenuItemWrapper.items()
.Not understanding how or why frustrates me some and my solution here may not address the real issue as a consequence. But this does work and it has me rolling and I can access the menus.
Also puzzling to me is that once I click the menu. The menu has one child, itself. And so I click it, extract that one item, and get its children. That works!