Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

disable option does not remove tests previously added to command line with -e option #213

Open
brad-kelly opened this issue May 9, 2023 · 1 comment

Comments

@brad-kelly
Copy link
Contributor

brad-kelly commented May 9, 2023

If I use the -e option to specify some tests to run, andthen explicitly disable some subset with the --disable option I would expect those test to be removed, but they are not.

Simple example:

% opendcdiag -v --output-format=tap -e zlib --disable=zlib
# opendcdiag -v --output-format=tap -e zlib --disable=zlib
# Operating system: Linux 5.15.0-46-generic, glibc 2.35
ok   1 zlib                     # (alpha test)   <-- marked alpha??
...

Why is this important (simple use case):
If I want to run all the eigen tests except those with "double" in the name, I'd expect to do something like:
opendcdiag -v --output-format=tap -e eigen* --disable=*double*

Unfortunately this does not work:

% opendcdiag -v  --output-format=tap -e eigen* --disable=*double*
# opendcdiag -v  --output-format=tap -e eigen* --disable=*double*
# Operating system: Linux 5.15.0-46-generic, glibc 2.35
ok   1 eigen_gemm_double14      # (alpha test) 
ok   2 eigen_gemm_cdouble_dynamic_square# (alpha test) 
ok   3 eigen_gemm_double_dynamic_square# (alpha test) 
ok   4 eigen_gemm_float_dynamic_square
ok   5 eigen_sparse
ok   6 eigen_svd
@busykai
Copy link
Contributor

busykai commented May 9, 2023

So right now what --disable does is not actually remove the test from the prepared list of tests to run, but sets its quality level to SKIP. This results on the (alpha test) remark in the output above. While this may work in some cases, it does not work in many others, such as when --disable tries to remove a test from the test list file, or exclude some of the tests are enabled using wildcard, or in a simple case when --enable and --disable specify the same test.

busykai added a commit that referenced this issue May 26, 2023
Or tests that are specified on the command line (either via -e or a test
list), but that are lower quality than requested.

Fixes #213.
busykai added a commit that referenced this issue May 26, 2023
Or tests that are specified on the command line (either via -e or a test
list), but that are lower quality than requested.

Fixes #213.

Signed-off-by: Arzhan Kinzhalin <arzhan.i.kinzhalin@intel.com>
busykai added a commit that referenced this issue May 26, 2023
Or tests that were specified on the command line (either via -e or a test
list), but that are lower quality than requested.

Fixes #213.

Signed-off-by: Arzhan Kinzhalin <arzhan.i.kinzhalin@intel.com>
busykai added a commit that referenced this issue May 26, 2023
Or tests that were specified on the command line (either via -e or a test
list), but that are lower quality than requested.

Fixes #213.

Signed-off-by: Arzhan Kinzhalin <arzhan.i.kinzhalin@intel.com>
busykai added a commit that referenced this issue May 26, 2023
Or tests that were specified on the command line (either via -e or a test
list), but that are lower quality than requested.

Fixes #213.

Signed-off-by: Arzhan Kinzhalin <arzhan.i.kinzhalin@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants