You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Since we change nengo tests over time, it's not uncommon that a backend passes one version of a test but not another. But currently backends can only indicate that a test passes or fails, there is no way to indicate that e.g. a test is expected to fail in 2.8.0 but pass in 3.0.0. As a specific example, we recently added the allclose fixture to let backends change tolerances on core tests. But that will only work in 3.0.0, backends won't be able to change the tolerances on 2.8.0 tests. So it would be nice if a backend could use the allclose functionality to get tests passing in versions that support that, and then just indicate that previous versions are expected to fail.
Describe the solution you'd like
We'd have to think about the syntax, but probably just some kind of special string added to the nengo_test_unsupported config, like
a/test/file.py:a_test
"{nengo_version<3.0} This test fails on Nengo < 3.0"
The text was updated successfully, but these errors were encountered:
I like this a lot, would solve a bunch of issues with keeping compatibility around just for testing.
Also, while we're at it, we currently get the Nengo test name through some weird file path logic, but we should switch to using the nodeid like we do in the Pytest fixtures for consistency (and because it makes more sense).
Is your feature request related to a problem? Please describe.
Since we change nengo tests over time, it's not uncommon that a backend passes one version of a test but not another. But currently backends can only indicate that a test passes or fails, there is no way to indicate that e.g. a test is expected to fail in 2.8.0 but pass in 3.0.0. As a specific example, we recently added the
allclose
fixture to let backends change tolerances on core tests. But that will only work in 3.0.0, backends won't be able to change the tolerances on 2.8.0 tests. So it would be nice if a backend could use theallclose
functionality to get tests passing in versions that support that, and then just indicate that previous versions are expected to fail.Describe the solution you'd like
We'd have to think about the syntax, but probably just some kind of special string added to the
nengo_test_unsupported
config, likeThe text was updated successfully, but these errors were encountered: