Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should size grow faster when the number of tests is small? #472

Open
ChickenProp opened this issue Dec 22, 2022 · 0 comments
Open

Should size grow faster when the number of tests is small? #472

ChickenProp opened this issue Dec 22, 2022 · 0 comments

Comments

@ChickenProp
Copy link
Contributor

I often don't want to run a full set of tests and use NoConfidenceTermination 10. (Specifically I use modifyMaxSuccess (const 10) from hspec-hedgehog). But that means the size never grows above 9, so Range.linear 0 10 will always generate 0, which isn't what I expected. It works out okay if I write my generators expecting a specific number of max-tests, or I can call Gen.scale everywhere. But it would be more convenient if the size would grow to something large in a small number of tests.

Quickcheck does something like this. I don't fully understand the arithmetic but in this case the size would go 0, 10, 20, ..., 90. (n is the test number and d is a discard count.)

I don't think it has an analogue of EarlyTermination or NoEarlyTermination, so hedgehog would need to figure out what to do about that. I don't use these myself, but from what I gather from the code:

  • NoEarlyTermination c t still runs exactly t tests, the same as NoConfidenceTermination t, so it could have the same behavior as NoConfidenceTermination.
  • EarlyTermination c t actually ignores t and runs at least 100 tests, more if needed for the confidence threshold c, but always some multiple of 100. (Is this intended? It's not what the name suggests to me.) So it could just keep the existing behavior.

So these don't seem like big problems to me.

ChickenProp added a commit to ChickenProp/haskell-hedgehog that referenced this issue Jan 30, 2023
ChickenProp added a commit to ChickenProp/haskell-hedgehog that referenced this issue Jan 30, 2023
Closes hedgehogqa#472. We now grow test size in fixed increments from 0 to 99, or
as close as we can without going over, in however many tests we run. If
we run more than n * 100 tests, then we go from 0 to 99 n times, and
then do fixed increments for the remainder. Additionally, if we discard
a bunch of times in a row we start to grow the size.
ChickenProp added a commit to ChickenProp/haskell-hedgehog that referenced this issue Mar 14, 2024
Closes hedgehogqa#472. We now grow test size in fixed increments from 0 to 99, or
as close as we can without going over, in however many tests we run. If
we run more than n * 100 tests, then we go from 0 to 99 n times, and
then do fixed increments for the remainder. Additionally, if we discard
a bunch of times in a row we start to grow the size.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant