Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to version 2 spec #17

Open
ErikSchierboom opened this issue Apr 8, 2021 · 1 comment
Open

Upgrade to version 2 spec #17

ErikSchierboom opened this issue Apr 8, 2021 · 1 comment
Labels
x:action/improve Improve existing functionality/content x:knowledge/advanced Comprehensive Exercism knowledge required x:module/test-runner Work on Test Runners x:size/large Large amount of work x:type/coding Write code that is not student-facing content (e.g. test-runners, generators, but not exercises)

Comments

@ErikSchierboom
Copy link
Member

ErikSchierboom commented Apr 8, 2021

We've recently updated the test runner interface specification to allow for two types of test runners, identified via a new version property in the results.json file (see the spec for the version property):

  1. version: 1 test runners are quite basic, and only detect if either all tests passed or not. These test runners capture the console output of the test runner and output that. They do not include details on individual tests passing/failing
  2. version: 2 test runners contain detailed information on individual tests passing/failing. We've recently added a required field named test_code, which contains the code the test ran to verify the behavior (see the spec for the test_code property).

The test runner's output is currently what I refer to as version 1.5: it contains the individual test information, but does not yet contain the test_code key. If possible, the test runner should be updated to also include the test code, in which case the version property should also be added with 2 as its value.

You are completely free on how you want to extract the test code, either via the AST or via finding the test code in the source code text via string manipulation.

Let me know if there are any questions.

@ErikSchierboom ErikSchierboom added x:action/improve Improve existing functionality/content x:knowledge/advanced Comprehensive Exercism knowledge required x:module/test-runner Work on Test Runners x:size/large Large amount of work x:type/coding Write code that is not student-facing content (e.g. test-runners, generators, but not exercises) labels Aug 17, 2021
@senekor
Copy link
Contributor

senekor commented Dec 15, 2023

I think this has to be done with AST parsing. Some test files use modules to organize the tests, I believe there is even one exercise where test cases are generated with a declarative macro.

The spec also says something about returning test results in the same order as the tests are defined. I believe this is currently not done correctly. Having AST information about the test file would be the perfect preparation to fix that as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
x:action/improve Improve existing functionality/content x:knowledge/advanced Comprehensive Exercism knowledge required x:module/test-runner Work on Test Runners x:size/large Large amount of work x:type/coding Write code that is not student-facing content (e.g. test-runners, generators, but not exercises)
Projects
None yet
Development

No branches or pull requests

2 participants