Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update slow tests marks and add script to automate #1164

Open
wants to merge 16 commits into
base: master
Choose a base branch
from

Conversation

matt-graham
Copy link
Collaborator

@matt-graham matt-graham commented Oct 16, 2023

Fixes #1145

Adds a new script src/scripts/automation/mark_slow_tests.py which uses a JSON report of Pytest results generated using pytest-json-report plugin to identify slow running tests and update the decorators used to mark the tests in the source code. The script uses RedBaron to parse the modules' source code in to a lossless abstract source tree, and adds or removes the slow mark decorators to the relevant test functions identified from the test report. To avoid too much switching back and forth for tests with duration close to the threshold used to determine if a test is 'slow' (currently ~10s), a basic form of hysteresis is used with slow marks only added if measured test duration is above a upper threshold (default 11s) and only removed if test duration is below a slower threshold (default 9s). A tox environment which install additional dependencies (pytest-json-report and redbaron), runs pytest with JSON report output and then runs mark_slow_tests.py script to update the tests is also added.

Updates to the slow marks based on a local run of the tests and script are also included in this PR - we could possibly separate this out in to a different PR. It would probably also be worth exploring automating running the script as an Actions workflow, potentially doing something like running on a weekly schedule and opening a PR for any changes to slow marks.

@matt-graham matt-graham marked this pull request as draft October 16, 2023 09:38
@matt-graham
Copy link
Collaborator Author

matt-graham commented Oct 16, 2023

Check failing due to not removing pytest import on modules for which all slow marks have been removed and no other pytest usages.

  • Add check if there are any remaining pytest usages after removing slow marks and remove pytest import if so

@matt-graham matt-graham marked this pull request as ready for review October 16, 2023 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Mark new slow running tests
1 participant