-
Notifications
You must be signed in to change notification settings - Fork 562
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tests: spread metadata used to filter and categorize tests #13971
base: master
Are you sure you want to change the base?
tests: spread metadata used to filter and categorize tests #13971
Conversation
This change introduces metadate on spread tests, this metadate is devided in features, groups and tags. In all cases the matadata can be used to filter tests to be executed in different circunstances, but also to know test coverage. The file spread.meta.yaml includes the values allowed to include in the tests. These values will be cross-checked in the ci to check the tests integrity.
ed30a79
to
7ef62b6
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, this looks like a really useful capability.
Some comments.
Furthermore:
- It might be useful to explain the envisioned rules of use e.g. can I filter by feature within a group?, filter by multiple features/groups etc.
- Do we perhaps need to cater for sub-features/granularity. If so, how to simplify this, do we perhaps want to consider a path-like notation?
Maybe we need a mini spec?
# of coverage documentation. | ||
|
||
definitions: | ||
# Features refer to product caracteristics being tested |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
characteristics
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps consider alternative definition of "features", perhaps something in the lines of:
"Features refer to specific product capabilities or sets of related capabilities."
|
||
definitions: | ||
# Features refer to product caracteristics being tested | ||
# The main porpoise is to know the features testing coverage |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
purpose
] | ||
|
||
# Groups are set of tests orthogonal to the suites definition | ||
# The main porpoise is to run an independent subset of tests to avoid test duplication |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
purpose
@@ -7,6 +7,11 @@ details: | | |||
snap set in the environment variables SPREAD_STORE_USER and SPREAD_STORE_PASSWORD, if | |||
they are not present then only the negative check is performed. | |||
|
|||
meta: | |||
features: [snap_refresh] | |||
groups: [sanity, core-validation] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
core-validation -> core_validation.
But perhaps worth asking, why "_" and not "-" which is perhaps more consistent with naming in spread?
@@ -3,6 +3,10 @@ summary: Check auto-refresh from a pre-download change | |||
details: | | |||
Verify that an inhibited auto-refresh triggers a pre-download change and resumes on close | |||
|
|||
meta: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not just leave it out?
snap_install # snap install command | ||
] | ||
|
||
# Groups are set of tests orthogonal to the suites definition |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps we can explain "orthogonal" in more practical terms e.g. set of tests that can be selected across multiple suites?
Assuming that's what is meant, is this not true to tags and features as well?
sanity # Minimal set of tests used to validate a system with more coverage than the smoke tests | ||
] | ||
|
||
# Tags are free labels used to identify group of tests |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wondering if the concept of tags is not similar enough to group that is not required as a separate concept?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
as I also discussed with @ernestl I think we need to have a mini specs to discuss what the different tag kinds are for, and naming conventions, initial listing for each of them
This change introduces metadata on spread tests, this metadata is divided in features, groups and tags.
In all cases the metadata can be used to filter tests to be executed in different circunstancies, but also to know test coverage.
The file spread.meta.yaml includes the values allowed to include in the tests. These values will be cross-checked in the ci to check the tests integrity.
The tests updated are just done as an example in order to understand how tests are going to looks like.