snap() captures whole page instead of element - can't replicate in Mac/Ubuntu

[yucongshub]

Hi! I am using RPA to capture the verification code image. It has always been normal when running on Windows. However, when I migrated the code to Linux for execution, I found that it did not capture the verification code image, but the entire page. Below is the test situation, Xpath matches it correctly, and I use r.click() it refreshes a new verification code, but if I use r.snap(), it does not screenshot the element, but the entire page.

View image 111.png, it is the entire page

My rpa and python versions are:
`
r.version
'1.50.0'

[root@wc-wuh-13-1-25-new zfw]# python3 -V
Python 3.10.10
`

I tried changing the chrome versions to 120, 108 and 73, but the problem still didn't solve. What could be the reason? How can I debug this?

[yucongshub]

I tested the site "baidu.com" and reproduced the problem.

Linux system:
r.snap('//img[ @id="s_lg_img" ]','222.png')
the result is the entire web page

Windows system:
r.snap('//img[ @id="s_lg_img" ]','222.png')
it is the correct element

[kensoh]

Hi @yucongshub thanks for raising this! The backend API only returns what Chrome returns. So my initial hunch is there might be some change or bug in Chrome in Linux that could be the root cause. And this is the first report of such issue.

I've tried replicating on Mac and it is also working fine -

RPA:Desktop kensoh$ python3
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpa as r
>>> r.init()
True
>>> r.url('www.baidu.com')
True
>>> r.snap('//img[ @id="s_lg_img" ]','222.png')
True

I have a Linux VPS but it is an old CentOS so I can't run Chrome there. Can you share with me the log file after running the snap() step from your Linux machine? I think you can directly attach the file here. It can be found by print(r.tagui_location()) and then go to .tagui/src/tagui_chrome.log

That is the transaction log between the TagUI engine and Chrome browser through websocket connection. So it contains the exact and entire low-level information returned from Chrome.

[kensoh]

Adding on @yucongshub - I've tried using the Google Colab notebook example (which runs on Ubuntu) and it appears to be working correctly. The notebook example link is the following and found on README.md page of this repo -

https://colab.research.google.com/drive/1or8DtXZP8ZxJYK52me0dA6O9A1dXKKOE?usp=sharing

Using the following lines, I have downloaded the log file and attaching it below for comparison later.

log_file = r.tagui_location() + '/.tagui/src/tagui_chrome.log'
from google.colab import files; files.download(log_file)

tagui_chrome.log

[yucongshub]

Hi @kensoh , I tested again and provided the previous and this time log files. I saw that the difference between the log files you provided is that there are no pixel coordinates in Page.captureScreenshot?

tagui_chrome.log.2024.1.15.log
tagui_chrome.log.2024.1.11.log

[yucongshub]

@kensoh I did some tests, using the coordinates returned in the tagui log, I used the following code to screenshot and it works fine on both Windows and Centos.

import rpa as r
import json
from websocket import create_connection
import base64
import requests

r.init()
r.url('https://baidu.com')
r.wait(2)

res = requests.get('http://localhost:9222/json').content
# print(json.loads(res.decode('utf-8'))[0]['webSocketDebuggerUrl'])


wsconn = json.loads(res.decode('utf-8'))[0]['webSocketDebuggerUrl']
ws = create_connection(wsconn,suppress_origin=True)

# r.snap('//img[ @id=\"s_lg_img\" ]','1111.png')

# centos
# [12] {"id":12,"result":{"result":{"type":"object","value":{"top":44,"left":526,"width":270,"height":129}}}}

# windows
# [8] {"id":8,"result":{"result":{"type":"object","value":{"top":44,"left":540,"width":270,"height":129}}}}

# do snap
# [9] {"id":9,"method":"Page.captureScreenshot","params":{"format":"png","quality":80,"clip":{"x":548,"y":44,"width":270,"height":129,"scale":1},"fromSurface":true}}

request = {}
request['id'] = 1
request['method'] = 'Page.captureScreenshot'
request['params'] = {"format":"png","quality":80,"clip":{"x":526,"y":44,"width":270,"height":129,"scale":1},"fromSurface":True}
ws.send(json.dumps(request))
result = ws.recv()
png = json.loads(result)['result']['data']
rpng = base64.b64decode(png)
# print(base64.b64decode(rpng))
open('testbaidu.png','wb').write(rpng)
ws.close()

r.wait(10)

How to continue troubleshooting this problem?

[kensoh]

Hi @yucongshub can you tell me more what do you mean? Can you share more on what is the difference between on your Linux and the example using Google Colab Ubuntu? Is it your Linux doesn't return coordinates and the Google Colab Ubuntu returns?

If on your Linux there is no coordinates returned, then you can't use workaround to capture the screenshot. Unless you use another system to first collect the coordinates and hard code the solution on your Linux system. Otherwise directly on Linux you don't have those coordinates to use the workaround.

What is your Linux and version? I suspect it might be some Linux/Chrome edge case issue that is hard to replicate on other Linux/Chrome. The solution might be to use the one you shared above, collect the coordinates on another machine and use them to do the screen shot on your Linux/Chrome.

[yucongshub]

Hi @kensoh , The Linux operating environment I use is centos7.9. I tested it in Windows, google colab, and centos7.9. Windows and google colab returned the correct element screenshots, but centos7.9 was wrong. Its coordinates and screenshot codes The execution order seems different from windows and google colab. How can I debug the order of obtaining element coordinates and taking screenshots? I want to compare and see how windows and centos7.9 run. I have provided the log files of the three, you can check from the logs.
tagui_chrome.log.google-colab.log
tagui_chrome.log.windows.log
tagui_chrome.log.centos7.log

[yucongshub]

Hi @kensoh , I recently changed my laptop system to ubuntu 24.04, I tested this problem again and it also recurred. I checked the logs and found something strange. The base64 encoding of the image is first output in the log, and then the coordinates are printed. This order is opposite to the windows and google-colab I provided before, and is the same as centos7 that also has the same problem. Is this the cause of this problem? ?

In addition, my centos7 and ubuntu24 have very few programs installed, and it is relatively clean, so I don’t think this is an accident, related to a specific third-party program or something like that.

tagui_chrome.log.ubuntu24.log