You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The default timeout of the derivation checks introduced in PR ?? causes inconsistent behavior when running the regression tests.
Since the timeout defaults to 5 seconds, the result of the derivation checks (success or timeout) depends on the performance of the system they are executed on.
The regression tests contain the following case studies with timed out derivation checks:
The first of these is the reason why the current regression tests fail. That is, the GitHub runner is fast enough such that the derivation check succeeds. The four other case studies are not part of the default regression tests.
The text was updated successfully, but these errors were encountered:
Hmm I guess to avoid this issue we need to either deactivate the derivation checks for most regression tests (except for those testing the derivation checks) to have stable results (in most cases), or to increment the timeout (which might cause longer run times, and might not always guarantee stable results). Any opinions?
The default timeout of the derivation checks introduced in PR ?? causes inconsistent behavior when running the regression tests.
Since the timeout defaults to 5 seconds, the result of the derivation checks (success or timeout) depends on the performance of the system they are executed on.
The regression tests contain the following case studies with timed out derivation checks:
case-studies-regression/fast-tests/related_work/YubiSecure_KS_STM12/Yubikey_and_YubiHSM_multiset_analyzed.spthy
case-studies-regression/sapic/fast/feature-secret-channel/U2F_analyzed.spthy
case-studies-regression/sapic/slow/PKCS11/pkcs11-templates_analyzed.spthy
case-studies-regression/ake/dh/DHKEA_NAXOS_C_eCK_PFS_partially_matching_analyzed.spthy
case-studies-regression/ake/dh/DHKEA_NAXOS_C_eCK_PFS_keyreg_partially_matching_analyzed.spthy
The first of these is the reason why the current regression tests fail. That is, the GitHub runner is fast enough such that the derivation check succeeds. The four other case studies are not part of the default regression tests.
The text was updated successfully, but these errors were encountered: