Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add new android operation(open app,exit,stop) #1207

Merged
merged 21 commits into from May 17, 2024
Merged

add new android operation(open app,exit,stop) #1207

merged 21 commits into from May 17, 2024

Conversation

kithib
Copy link
Contributor

@kithib kithib commented Apr 18, 2024

Features

  • xx
  • yy

Feature Docs

Influence

You can use EXIT to return to the main interface of the mobile phone and turn off the application that is running in the background
You can use stop to return to the task success information
You can use OpenApp without XML to turn on the mobile app
You can use Click Text without XML to click special text icon
Result
截屏2024-04-18 11 57 39

Other

截屏2024-04-18 11 57 19 截屏2024-04-18 15 30 13 截屏2024-04-18 16 03 14

@geekan
Copy link
Owner

geekan commented Apr 22, 2024

/review

Copy link

PR Review

⏱️ Estimated effort to review [1-5]

4, due to the extensive changes across multiple files involving complex functionalities like OCR, image processing, and interaction with external services. The PR integrates multiple new libraries and modifies critical functionalities, requiring a thorough review to ensure compatibility and performance.

🧪 Relevant tests

No

🔍 Possible issues

Possible Bug: The method user_open_app in android_ext_env.py might fail silently if no app name matches the OCR result. This could lead to unexpected behavior without proper error handling or feedback to the caller.

Performance Concern: The user_click_icon method involves downloading a model file every time it is called, which could significantly affect performance and reliability, especially in a production environment.

🔒 Security concerns

No

Code feedback:
relevant filemetagpt/environment/android/android_ext_env.py
suggestion      

Consider adding error handling or a retry mechanism in the user_open_app method to manage cases where the OCR fails to detect the specified app name. This will improve the robustness of the application. [important]

relevant linelogger.info(f"No App named {app_name}.")

relevant filemetagpt/environment/android/android_ext_env.py
suggestion      

To improve performance, consider caching the model file locally after the first download in the user_click_icon method, instead of downloading it every time the method is called. This change would reduce latency and bandwidth usage. [important]

relevant linefile_path = download_model(file_url, target_folder)

relevant filemetagpt/environment/android/android_ext_env.py
suggestion      

Implement a cleanup mechanism for temporary files created during the icon detection process in the user_click_icon method to prevent storage overflow and potential security risks from stale data. [medium]

relevant linecrop_image = f"{i}.jpg"

relevant filemetagpt/environment/android/android_ext_env.py
suggestion      

Optimize the sleep intervals in get_screenshot method to be configurable or dynamically adjusted based on device response times, which could enhance the responsiveness and efficiency of screen capture operations. [medium]

relevant linetime.sleep(0.1)


✨ Review tool usage guide:

Overview:
The review tool scans the PR code changes, and generates a PR review which includes several types of feedbacks, such as possible PR issues, security threats and relevant test in the PR. More feedbacks can be added by configuring the tool.

The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.

  • When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:
/review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...
[pr_reviewer]
some_config1=...
some_config2=...

See the review usage page for a comprehensive guide on using this tool.

return exit_res

def _ocr_text(self, text: str) -> list:
if not self.screenshot_dir.exists():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

already use self.create_device_path. no need.


@mark_as_writeable
def user_click_icon(self, icon_shape_color: str) -> str:
if not self.screenshot_dir.exists():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same.

@better629
Copy link
Collaborator

lgtm

@geekan geekan merged commit d9ed99e into geekan:main May 17, 2024
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants