Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recreates OmniFixture dump used for UI tests #22170

Merged
merged 7 commits into from May 7, 2024
Merged

Conversation

sgiehl
Copy link
Member

@sgiehl sgiehl commented Apr 29, 2024

Description:

Many of our UI tests are using a prepared database dump to set up their fixture. This dump was last recreated somewhen before release of Matomo 4. So every test actually first performes an update from Matomo 3.16 to Matomo 5.
In addition that means that some data might not even be available for tests, like values for dimensions that had been added after the dump had been created.

This PR recreates the database dump and updates all screenshots accordingly.

There are a lot of changes in the screenshots. This is caused by the way our OmniFixture is created. It basically sets up all test fixtures that are defined in core tests at once. Due to many changes in such fixtures since Matomo 3.16, a lot of report data has changed now. I've looked through all the changes and too everything looked like a reasonable change in displayed report data only.

Some resulting screenshots in Treemap plugin also will need to be updated before merging this one.

Note: This will be required to run UI tests on TiDb

Review

@michalkleiner
Copy link
Contributor

Just out of curiosity, do we have some documentation on how to recreate the dump / omni fixture from a newer version?

@sgiehl
Copy link
Member Author

sgiehl commented Apr 30, 2024

@michalkleiner Yes. That's documented here: https://developer.matomo.org/guides/tests-system#omnifixture

@sgiehl sgiehl added this to the 5.1.0 milestone Apr 30, 2024
@sgiehl sgiehl added c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. Needs Review PRs that need a code review labels Apr 30, 2024
@sgiehl sgiehl requested a review from a team May 2, 2024 07:53
@sgiehl sgiehl marked this pull request as ready for review May 2, 2024 07:53
@sgiehl sgiehl mentioned this pull request May 2, 2024
11 tasks
Copy link
Member

@mneudert mneudert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to give anyone a heads-up before merging this so they are aware that their test results could change?

@sgiehl
Copy link
Member Author

sgiehl commented May 3, 2024

@mneudert Any UI test that does not specify an own fixture by default uses this one. So it might affect certain plugins...

@sgiehl
Copy link
Member Author

sgiehl commented May 6, 2024

@matomo-org/plugin-reviewers We are going to update the dataset of UITestFixture with this PR. Any plugin UI test that relies on this fixture, might afterwards display different data, as there had been a lot changes since the fixture had been last updated. So if there are any tests not using a custom fixture, they might start failing afterwards.

@sgiehl sgiehl merged commit 1845633 into 5.x-dev May 7, 2024
22 of 25 checks passed
@sgiehl sgiehl deleted the omnifixtureupdate branch May 7, 2024 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. Needs Review PRs that need a code review not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants