Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Many tests are skipped in test-ref-configs #9063

Open
gilles-peskine-arm opened this issue Apr 26, 2024 · 2 comments · May be fixed by #9149
Open

Many tests are skipped in test-ref-configs #9063

gilles-peskine-arm opened this issue Apr 26, 2024 · 2 comments · May be fixed by #9149
Labels
bug component-test Test framework and CI scripts size-m Estimated task size: medium (~1w)

Comments

@gilles-peskine-arm
Copy link
Contributor

gilles-peskine-arm commented Apr 26, 2024

Looking at the logs of component_test_ref_configs on development, a lot of intended test cases are not executed. Contrast with 2.28. For example, with config-thread.h, which focuses on ECJPAKE, both in legacy and PSA mode:

 running ssl-opt.sh -f ECJPAKE.*nolog (config-thread.h)
 ECJPAKE: working, DTLS, nolog .......................................... SKIP
 ------------------------------------------------------------------------
 PASSED (1 / 1 tests (1 skipped))

This test case should pass, and does pass in 2.28.

A full log of all.sh test_ref_configs on 771fd7d, for convenience and to serve as a reference point: test-ref-configs-771fd7d1dcd19a157812dde9766bb7781e60d075.txt

The goal of this issue is to:

  • Figure out what's going wrong, and possibly since when.
  • Propose a method to determine whether the intended features are tested in each configuration.
  • If that method involves tooling, develop that tooling.
  • Fix all the bugs, i.e. arrange for each configuration's tests to actually run the intended tests.
  • Possibly design and implement a way to ensure this doesn't happen again.

This may be broken up into sub-issues depending on what the initial analysis finds.

@gilles-peskine-arm gilles-peskine-arm added bug size-m Estimated task size: medium (~1w) component-test Test framework and CI scripts labels Apr 26, 2024
@gilles-peskine-arm
Copy link
Contributor Author

At least one of the problems is #8366. But I think it's not the only one, I'll need to read the logs further.

@gilles-peskine-arm
Copy link
Contributor Author

Possibly design and implement a way to ensure this doesn't happen again.

So far, we like the idea that ssl-opt.sh and compat.sh will, by default, complain if all test cases are skipped. (private discussion). This is far from perfect, but it's very easy to do, so it has a high efficacy/cost ratio.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug component-test Test framework and CI scripts size-m Estimated task size: medium (~1w)
Projects
EPICs for Mbed TLS
Test cases not executed
Status: No status
Status: No status
Development

Successfully merging a pull request may close this issue.

1 participant