Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

INTERPRETING.md should clarify that PASS/FAIL results are per test (file), not scenario (strict, non-strict) #3871

Open
linusg opened this issue Jul 15, 2023 · 2 comments
Labels
valid and awaiting action Issue is valid, but no action taken yet

Comments

@linusg
Copy link
Member

linusg commented Jul 15, 2023

The concept of a "test scenario" is not outlined in INTERPRETING.md but common in runners such as https://github.com/bocoup/test262-stream and https://github.com/bterlson/test262-harness, which reports separate PASS and FAIL results per test scenario (approx. twice the test file count, ignoring noStrict and onlyStrict). This is problematic as it can lead to test results that are skewed towards a more positive outcome than what I believe is the intended way of counting test results: a "test" (one file) only passes if, by default, the outcome matches the expectation both in strict and non-strict mode.
A hypothetical engine failing all tests in strict mode and passing all in non-strict mode would therefore receive a score of 0%, not 50% of "passing tests".

@linusg
Copy link
Member Author

linusg commented Jul 15, 2023

Relevant sections from INTREPRETING.md that allow to infer this conclusion but could be more explicit:

All tests are declared as text files located within this project's test directory.

Unless configured otherwise (via the noStrict, onlyStrict, module, or raw flags), each test must be executed twice: once in ECMAScript's non-strict mode, and again in ECMAScript's strict mode.

By default, tests signal failure by generating an uncaught exception. If execution completes without generating an exception, the test must be interpreted as "passing." Any uncaught exception must be interpreted as test failure.

@ljharb
Copy link
Member

ljharb commented Jul 15, 2023

I would certainly not expect partial credit for only implementing one of strict or non-strict, so clarifying the docs seems fine with me.

@ptomato ptomato added the valid and awaiting action Issue is valid, but no action taken yet label Aug 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
valid and awaiting action Issue is valid, but no action taken yet
Projects
None yet
Development

No branches or pull requests

3 participants