tests: spread metadata used to filter and categorize tests #13971
Conversation
This change introduces metadate on spread tests, this metadate is devided in features, groups and tags. In all cases the matadata can be used to filter tests to be executed in different circunstances, but also to know test coverage. The file spread.meta.yaml includes the values allowed to include in the tests. These values will be cross-checked in the ci to check the tests integrity.
sergiocazzolato
force-pushed
the
tests-new-spread-metadata
branch
from
ed30a79
to
7ef62b6
Compare
There was a problem hiding this comment.
Thanks, this looks like a really useful capability.
Some comments.
Furthermore:
- It might be useful to explain the envisioned rules of use e.g. can I filter by feature within a group?, filter by multiple features/groups etc.
- Do we perhaps need to cater for sub-features/granularity. If so, how to simplify this, do we perhaps want to consider a path-like notation?
Maybe we need a mini spec?
| # of coverage documentation. | ||
|
|
||
| definitions: | ||
| # Features refer to product caracteristics being tested |
There was a problem hiding this comment.
Perhaps consider alternative definition of "features", perhaps something in the lines of:
"Features refer to specific product capabilities or sets of related capabilities."
|
|
||
| definitions: | ||
| # Features refer to product caracteristics being tested | ||
| # The main porpoise is to know the features testing coverage |
| ] | ||
|
|
||
| # Groups are set of tests orthogonal to the suites definition | ||
| # The main porpoise is to run an independent subset of tests to avoid test duplication |
| @@ -7,6 +7,11 @@ details: | | |||
| snap set in the environment variables SPREAD_STORE_USER and SPREAD_STORE_PASSWORD, if | |||
| they are not present then only the negative check is performed. | |||
|
|
|||
| meta: | |||
| features: [snap_refresh] | |||
| groups: [sanity, core-validation] | |||
There was a problem hiding this comment.
core-validation -> core_validation.
But perhaps worth asking, why "_" and not "-" which is perhaps more consistent with naming in spread?
| @@ -3,6 +3,10 @@ summary: Check auto-refresh from a pre-download change | |||
| details: | | |||
| Verify that an inhibited auto-refresh triggers a pre-download change and resumes on close | |||
|
|
|||
| meta: | |||
There was a problem hiding this comment.
Why not just leave it out?
| snap_install # snap install command | ||
| ] | ||
|
|
||
| # Groups are set of tests orthogonal to the suites definition |
There was a problem hiding this comment.
Perhaps we can explain "orthogonal" in more practical terms e.g. set of tests that can be selected across multiple suites?
Assuming that's what is meant, is this not true to tags and features as well?
| sanity # Minimal set of tests used to validate a system with more coverage than the smoke tests | ||
| ] | ||
|
|
||
| # Tags are free labels used to identify group of tests |
There was a problem hiding this comment.
Wondering if the concept of tags is not similar enough to group that is not required as a separate concept?
pedronis
added
the
Needs Samuele review
pedronis
added
⛔ Blocked
and removed
Simple 😃
This change introduces metadata on spread tests, this metadata is divided in features, groups and tags.
In all cases the metadata can be used to filter tests to be executed in different circunstancies, but also to know test coverage.
The file spread.meta.yaml includes the values allowed to include in the tests. These values will be cross-checked in the ci to check the tests integrity.
The tests updated are just done as an example in order to understand how tests are going to looks like.