Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverage checking of state machines is awkward #471

Open
ChickenProp opened this issue Dec 2, 2022 · 0 comments
Open

Coverage checking of state machines is awkward #471

ChickenProp opened this issue Dec 2, 2022 · 0 comments

Comments

@ChickenProp
Copy link
Contributor

ChickenProp commented Dec 2, 2022

#469 reminded me of this.

So currently, there's a a difference between these two:

cover 10 "condition" condition
when condition $ cover 10 "condition" True

because if condition is never true, the former will fail while the latter will pass. (The label's never been seen, Hedgehog doesn't know it should have 10% coverage.) For the most part that feels like not a big deal.

But afaict it doesn't work well with state machines. I think the natural way to say "I want this command to run in 10% of passes" is to put cover 10 "command" True in the Ensure. But then you'll only get a failure if it runs less than 10% of the time but still at least once.

I've been getting around this by doing label "command" in the Ensure, and cover 10 "command" False after executeSequential. But this relies on internal details: label n is cover 0 n True, the semigroup instances for Label and Coverage are right-biased, and the semigroup instance for Cover is Cover-biased. So a test doing label "name" followed by cover 10 "name" False is equivalent to cover 10 "name" True, while doing it in reverse would be cover 0 "name" True; and if different test runs give different coverage requirements for the same name, the one from the last test run is what matters.

Unfortunately, this also means the failure report points at the cover line, not the label line. (If you put cover before executeSequential, it would fix that but no longer work reliably. It would be fine if the command in question didn't execute in in the final test run. But if it did, it would have no required coverage.)

I think a reasonable way to fix this would be for labelMinimum to be a Maybe CoverPercentage. cover would set that to Just while label, classify, and collect would set it to Nothing. The semigroup instance for Label would still take labelName and labelLocation from the second argument, but labelMinimum would take a Just from the first over a Nothing from the second. We'd need to document that that's how they interact, but I think it's a reasonable way for them to interact.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant