Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mark unsupported tests by version #1565

Open
drasmuss opened this issue Aug 16, 2019 · 1 comment
Open

Mark unsupported tests by version #1565

drasmuss opened this issue Aug 16, 2019 · 1 comment

Comments

@drasmuss
Copy link
Member

drasmuss commented Aug 16, 2019

Is your feature request related to a problem? Please describe.

Since we change nengo tests over time, it's not uncommon that a backend passes one version of a test but not another. But currently backends can only indicate that a test passes or fails, there is no way to indicate that e.g. a test is expected to fail in 2.8.0 but pass in 3.0.0. As a specific example, we recently added the allclose fixture to let backends change tolerances on core tests. But that will only work in 3.0.0, backends won't be able to change the tolerances on 2.8.0 tests. So it would be nice if a backend could use the allclose functionality to get tests passing in versions that support that, and then just indicate that previous versions are expected to fail.

Describe the solution you'd like

We'd have to think about the syntax, but probably just some kind of special string added to the nengo_test_unsupported config, like

a/test/file.py:a_test
  "{nengo_version<3.0} This test fails on Nengo < 3.0"
@tbekolay
Copy link
Member

I like this a lot, would solve a bunch of issues with keeping compatibility around just for testing.

Also, while we're at it, we currently get the Nengo test name through some weird file path logic, but we should switch to using the nodeid like we do in the Pytest fixtures for consistency (and because it makes more sense).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants