Announcement: I'm making a PyWinAutoUI (graphical) GUI with advanced features

[enjoysmath]

My end goal is to do on-screen lessons with overlayed arrows as done on bubble.is (and their set of lessons), but since I don't want to always be coding to create a simple enough lesson, I thought I should give PyWinAuto a GUI frontend that detects / records clicks in .py script format.

Here's a screenshot:

I'm assuming I should post back here when it's ready.

[vasily-v-ryabov]

Hi @enjoysmath nice to see efforts to extend pywinauto ecosystem. We're open to all new ideas and initiatives.

Maybe it's worth telling you about our own efforts:

• py_inspect is a simple GUI inspect tool which is only 150 lines of code using PyQt5. We hope to extend it with advanced recording features in the future, but at the moment there are no people who is ready to do this job.
• Automatic script generator is a command line tool that uses all possible hooks and event subscriptions to record your actions into a pywinauto script with text properties where possible. This is not a stupid coordinate recorder/player, but a generator of reliable and maintainable scripts. Unfortunately this 2-years work is paused, because we want to finish Linux and macOS support as well as some re-factoring to make the whole library interface more simple and consistent.

The second tool could be integrated into py_inspect GUI with some additions: single step recording, Undo/Redo and other recorded script editor features. It would be nice to discuss all these things with community.

[enjoysmath]

@vasily-v-ryabov I have already integrated PyInspect efforts into my code. Unfortunately, I'm new at github and don't know how to recover an old repository (because currently pywinauto/recorder.py is not present). I have already started on a recorder as well, but it would be nice to use an official one coded by someone who isn't a complete newbie at pywinauto :) Especially because now I have to move the recorder into a separate process because it crashes the whole app if you try to quit the thread in a simple hook and listen recorder.

Here's a screenshot:

@vasily-v-ryabov Correct me if I'm wrong, but a simple google search turned up and I should use:

https://github.com/pywinauto/pywinauto-sandbox/blob/master/sandbox/recorder.py

I will look into using that in a separate thread first, and if it crashes my app, in a different process.

[enjoysmath]

Here's another screenshot showing progress:

I was able to use the multiprocessing library and run the recording on a separate process. So sockets are used to communicate back to the GUI process. It's running without blocking the GUI and doesn't seem to crash anything now. Now just have to convert the events into code statements.

I didn't use the sandbox/recorder.py completely since I was using pywinauto.win32_hooks library while the recorder is old and uses pyHook which has to be replaced by pyWinhook. I might have an option to select which one to use. I will use the existing recorder code to inspire the translation to code calls of each of the events.

I want to release the source (because I have to according to PyQt5 open source license) and sell binaries with a selected time period of updates + Jira user account on bitbucket issue forums. Not for much though, maybe $5-10 / license? The source code can of course be downloaded & run, but I would be relying on the fact that most people don't want to go through the headache of doing that. I'm hesitant to sell the software, since the backend library (pywinauto) was free for me to use, but on the other hand no one is giving me work, so... not sure what to do.

[vasily-v-ryabov]

No, sandbox repository is very very old! Please don't use it.
I've already posted the link to pull request which contains instructions and code difference:
#701

[vasily-v-ryabov]

Regarding selling the binaries... of course, you're free to use pywinauto core library for commercial purpose since all the licenses permit it. No problem about that. We can't sponsor this work as there are no donations from community last year. So I completely understand your wish.

The only thing I take care of is a license of your source code. If it's GPL, we can't integrate it into pywinauto core library or into py_inspect. So we will have to re-implement it on our side. I prefer BSD 3-clause or MIT license, but in some cases LGPL might be OK.

If you were so glad to provide pull requests to py_inspect repository, that would be really cool.

[vasily-v-ryabov]

Hope it's easy to clone @cetygamer 's fork repository and his branch with the recorder to try it out (just click at the source branch in the pull request #701). Feel free to ask me about the architecture of the recorder.

[vasily-v-ryabov]

Sorry for delayed replies. I have a good main job and we work remotely for this quarantine period. So I have no commercial interest for this project in nearest years.

I will monitor this issue in a high priority order and can promise to reply once a day. So that you can spend your time in the most efficient direction.

And thanks for all your efforts!

[enjoysmath]

@vasily-v-ryabov Thank you for all of your prompt responses!

I'm not sure how to do a pull request for py_inspect. Also I totally broke up py_inspect's code into modules and it runs on a thread as well. I'm not sure how doing a pull request applies here.

According to this FAQ: https://www.riverbankcomputing.com/commercial/license-faq
I am limited by PyQt5 library to GPL.

According to this: https://www.riverbankcomputing.com/commercial/buy
PyQt5 is only $550 but you have to also purchase Qt5 which is over $5000: https://www.qt.io/buy-product/. So I guess I will be going open source on this one, plus commercial binaries with support.

From your link I clicked on the 'Commits' tab and found these goodies: e1efd6a

That looks like a lot of code to re-invent, so I'll try to put it into PyWinAutoUI.

Found this which is even nicer:
https://github.com/pywinauto/pywinauto/tree/19867af80449f9e4448bebc2b083ee584f0fbff4/pywinauto/recorder

[enjoysmath]

@vasily-v-ryabov It would be crazy to try to get that old code to work. I cannot seem to bring up that branch in TortoiseGit. So unless you can provide me with a zip file of or method of checking out code that would plug into the most recent version of pywinauto, that approach is ruled out.

I am happy with a simpler approach that I can code from scratch (using latest release of pywinauto) and that will meet my needs. My end goal is a set of on-screen lessons using the apps Qt Creator, WingWare, and Chrome browser, and PyQt5. By this I mean the user is directed where to click, where to type etc (like bubble.is lessons). The lessons will be created using a "Create Lesson" feature of PyWinAutoUI.

[vasily-v-ryabov]

You can fork cetygamer/pywinauto repository to your GitHub account by pressing "Fork" and waiting for a few minutes.

Then you can clone the repo like this (just replace "cetygamer" with "enjoysmath") in cmd.exe or in GitBash:

git clone --branch dev-script-generator https://github.com/cetygamer/pywinauto.git .\pywinauto_fork

After the repo is cloned let's change working directory to it and try recorder:

cd .\pywinauto_fork
python.exe .\pywinauto\recorder\recorder_cli.py -b uia --cmd .\apps\WPF_samples\WpfApplication1.exe -v -o recorded_wpf.py

Some steps might be not recorded with tracebacks in console because it is still prototype. For example, combo box actions is one of the complicated actions that were not finished. But some simple actions are recorded. Actions on side windows (not in the target app) are not recorded.

[enjoysmath]

@vasily-v-ryabov Thank you for helping with git and what to git. I've got it running and it's throwing some errors, probably because I'm using Python 3.8. I will get it through the errors.

Question: why do they target a single application when doing the recording?

I was able to do pywinauto.backend.registry.backends["uia"].element_info_class() or something similar to get the "root of all things" for which I was also able to record clicks. If targeting specific apps is best, I can code it to target a small set of apps.

[enjoysmath]

@vasily-v-ryabov should I work from cetygamer's or pywinauto's repo to get the recorder up-and-running again?

[enjoysmath]

Another question: I'm changing .. to pywinauto in a lot of places, usually this is because the way I'm running the source file is not correct. So what is the way in which one normally runs the recorder say from a pywinauto user's perspective? That is what I need since these import changes will likely end up breaking things when I switch to the standard way.

OTOH, maybe there is a quick fix for these import errors:

There's about 100 or so of these errors I'm estimating because they halt compilation. This is with
running from WingWare IDE -or- the command line (same errors), using the command line you gave.

Update

Fixed the error by prepending to sys.path:
sys.path = [os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..")] + sys.path
on that first line above the imports. On to next error I think :)

Update1

Now it's running using the command line you gave & from the command line, but not from within my WingWare IDE.

I have to try to get it to know about the pywinauto folder where apps is located.

Update2

Got it to work from IDE with absolute path passed in:
-b uia --cmd "C:\Users\FruitfulApproach\Desktop\pywinauto_fork\apps\WPF_samples\WpfApplication1.exe" -v -o recorded_wpf.py
So before calling this CLI you just convert the path using os.path.abspath(). I tried converting it after the point of the CLI's config option parsing but something else is causing it not to find it. Thus the obvious solution is just to always pass in the absolute path here.

Update3

This is awesome! It generated the following script:

# encoding: utf-8
import os, sys
script_dir = os.path.dirname(__file__)
sys.path.append(script_dir)
import pywinauto
recorded_version = '0.6.6'
print('Recorded with pywinauto-{}'.format(recorded_version))
print('Running with pywinauto-{}'.format(pywinauto.__version__))

app = pywinauto.Application(backend='uia').start(r'C:\Users\FruitfulApproach\Desktop\pywinauto_fork\apps\WPF_samples\WpfApplication1.exe')
app.WPFSampleApplicationDialog.button3Button.invoke()
app.WPFSampleApplicationDialog.TabControlTreeandListViews.select('ListBox and Grid')
app.WPFSampleApplicationDialog.TitleBar.click_input(button='left', coords=(256, 12))
app.WPFSampleApplicationDialog.TabControlTreeandListViews.select('Tree and List Views')
app.WPFSampleApplicationDialog.TitleBar.click_input(button='left', coords=(312, 7))
app.WPFSampleApplicationDialog.WpfApplication1LvItemDataItem.click_input(button='left', coords=(232, 9))
app.WPFSampleApplicationDialog.TabControlTreeandListViews.click_input(button='left', coords=(428, 238))
app.WPFSampleApplicationDialog.TitleBar.click_input(button='left', coords=(330, 28))
app.WPFSampleApplicationDialog.MondayStatic.click_input(button='left', coords=(9, 8))
app.WPFSampleApplicationDialog.TabControlTreeandListViews.click_input(button='left', coords=(423, 236))
app.WPFSampleApplicationDialog.MaximizeButton.click_input(button='left', coords=(10, 12))
app.kill()
app.kill()

[enjoysmath]

@vasily-v-ryabov I'm not sure what to do next as far as getting this to work with the latest pywinauto. I'll assume I just fork pywinauto/pywinauto and begin manually migrating the recorder code over from cetygamer/pywinauto fork. I think this is a first step, as it would be weird to depend on both repositories and there would be conflicts involving the generated code.

[enjoysmath]

@vasily-v-ryabov
Any idea how to fix this error:

I made minimal changes after copying over the recorder folder to other parts of the code in order to get it to run. See master branch of: https://github.com/enjoysmath/pywinauto

Update

I fixed that line by using GetModuleHandleW instead of GetModuleHandleA but I wrapped the original in a try/except block so it should work as original code except if in my case it breaks then it will use the unicode version (the 'W' version) of GetModuleHandle func. Again, see master branch of my forked repo. I'm still not sure how to link you to the changes made.

Had to make a __contains__ method for ControlTreeNode:

    def __contains__(self, point):
        try:
            if point in self.rect:
                return True
        except TypeError:
            r = self.rect
            
            if isinstance(point, tuple):
                x = point[0]
                y = point[1]
            else:
                x = point.x
                y = point.y
                
            if x >= r.left and y >= r.top and x <= r.right and y <= r.bottom:
                return True
        return False

because with all my changes made there were then two versions of a rectangle struct floating around. This will work with both versions of the rect. So instead of point in node.rect you use point in node. Hope that's okay.

Update1

I did git uninstally pywinauto and then git install git+https://github.com/enjoysmath/pywinauto.git so that my code now uses my repo. When can we get things merged?

I ran my PyWinAutoUI code and it crashes once with a weird error but upon running again everything runs as expected. I don't know if that was an environment fluke or what involving my IDE's setup. But anyway, everything seems to be working okay at least with my test case. Perhaps you could show me how to run tests on pywinauto so that I make sure I don't break anything.

[airelil]

I fixed that line by using GetModuleHandleW instead of GetModuleHandleA but I wrapped the original in a try/except block so it should work as original code except if in my case it breaks then it will use the unicode version (the 'W' version) of GetModuleHandle func.

Was fixed in another branch: 201bd21

[enjoysmath]

I fixed that line by using GetModuleHandleW instead of GetModuleHandleA but I wrapped the original in a try/except block so it should work as original code except if in my case it breaks then it will use the unicode version (the 'W' version) of GetModuleHandle func.

Was fixed in another branch: 201bd21

How did you find this out?

[enjoysmath]

@airelil I forked and cloned pywinauto/pywinauto so if it was fixed and atspi is the default branch, then why didn't my checked out code show that updated code? So, how would I bring in the changes to my code using git checkout?

[airelil]

As I understand, you worked on a "recorder" branch that probably is quite behind the rest of the code. This is a complicated feature and the work on it has been done in parallel by other contributors. Usually, once the feature is stable enough it's merged into the main branch. Currently we have master as our stable branch and atspi as our development branch where we consolidate all upcoming features for the next release. To align with atspi branch you need to run "git merge", but at this stage with such a big difference between two branches (many files have been moved under different locations) it will require to resolve many conflicts in different areas of the code.

[airelil]

BTW, @enjoysmath, did you manage to run unit tests?
Edit: I see you already created #909 so your are able to execute it

[vasily-v-ryabov]

@enjoysmath sorry I was unable to be near PC during the weekend. I'm going to merge atspi branch into dev-script-generator. This is pretty complicated merge because current state of recorder is based on master branch while atspi has some deep changes with moving the code to OS specific modules (master branch is Windows only). Also I will push the recorder branch to the upstream (pywinauto/pywinauto). Hope it will make your life simpler.

[vasily-v-ryabov]

I've almost finished the merge in my fork. Will push it to upstream when the tests are passed.

At the same time it is useful to learn how to add remote repositories to the local clone and sync from another fork or from upstream:
https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/syncing-a-fork

[vasily-v-ryabov]

Well, the updated "atspi-based" branch in my fork looks in the same state as "master-based" branch:

• AppVeyor tests for master-based branch
• AppVeyor tests for atspi-based branch

I've done for today. The tests are still having few failures. But I've pushed the branch to upstream as well. Can't promise to fix them quickly. But it's possible to send pull requests to this branch at least. ;)

[vasily-v-ryabov]

Regarding several applications monitoring... of course, it's possible. We didn't think in details about the whole mechanism, but we plan to implement spawning child processes detection in Application object in the core library. We may think to add a list of processes or handle several Application objects at the same time. This is interesting point to discuss.

[vasily-v-ryabov]

Not sure about dependencies between py_inspect and pywinauto. I thought it could be integrated so: py_inspect uses pywinauto as a package installed into Python. But temporary prototype could be implemented in the way you like. The command line recorder is planned to be a part of pywinauto repository to avoid maintaining versions compatibility between recorder and core library.

[jjbright]

This looks interesting to me, can I know how to use the code generator for pywinauto and how to run the pywinauto GUI @enjoysmath @vasily-v-ryabov @airelil
I'm struggling a lot using the print_control_identifiers() and find the path for the controls in the screen.

The control types are UIA

[enjoysmath]

@jjbright Follow the instructions here (above):

If you want to run the debugger from the commandline you have to specify the full absolute path of the executable app.

[jjbright]

@jjbright Follow the instructions here (above):

If you want to run the debugger from the commandline you have to specify the full absolute path of the executable app.

I tried like below
`
C:\Python\Python37>python .\pywinauto\recorder\recorder_cli.py -b uia --cmd C:\Windows\System32\notepad.exe -v -o recorded_wpf.py
python: can't open file '.\pywinauto\recorder\recorder_cli.py': [Errno 2] No such file or directory

C:\Python\Python37>
`

I've python 3.7 64bit
pywinauto 0.6.8

[enjoysmath]

@jjbright Did you git the source code of that other repository? The recorder code is not part of this repo afaik. (See instructions again)

[jjbright]

@jjbright Did you git the source code of that other repository? The recorder code is not part of this repo afaik. (See instructions again)

@enjoysmath I followed the same steps as mentioned, so now I get ModuleNotFoundError

C:\Users\jjbright\source\repos\pywinauto_fork>python.exe .\pywinauto\recorder\recorder_cli.py -b uia --cmd C:\Windows\System32\notepad.exe -v -o recorded_wpf.py Traceback (most recent call last): File ".\pywinauto\recorder\recorder_cli.py", line 17, in <module> from pywinauto.recorder.uia.uia_recorder import UiaRecorder ModuleNotFoundError: No module named 'pywinauto.recorder'