TimeBox a Property test

[dharmaturtle]

I'm building a TimeboxedPropertyAttribute for Hedgehog.Xunit. Its documentation:

Runs tests until the time box is reached or the tests parameter is hit (100 by default). If no size is provided, a random size will be used for each test run.

Raison d'être: I have some tests that take a while to run. During development, I don't care that much about correctness and will therefore timebox said tests. I'll flip them back to normal property tests before committing/merging. Since a test might only run once if the timebox is extremely short, I want a random size. (Always testing with Size=0 isn't great.) This is pretty niche to my workflow, but LMK if any of this sounds interesting and anyone wants it merged upstream.

---

I'm currently implementing it via a hack (but without reflection):

  ...
  let reportWith config =
    Property.forAll invoke
    >> match recheck with
       | Some (size, seed) -> Property.reportRecheckWith size seed config
       | None              -> Property.reportWith config
  match timeBox with
  | Some timeBox ->
    let config =
      PropertyConfig.defaultConfig
      |> PropertyConfig.withTests 1<tests>
      |> withShrinks shrinks
    let tests = tests |> Option.defaultValue 100<tests>
    let rng = Random()
    let report () =
      101
      |> rng.Next
      |> Gen.resize
      |> gens
      |> reportWith config
    let stop = DateTime.UtcNow + timeBox
    let rec loop i r =
      if DateTime.UtcNow >= stop || i >= tests || isNotOk r then
        { r with Tests = i }
      else
        report() |> loop (i + 1<tests>)
    report() |> loop 1<tests>
  ...

Basically, I run the test once, see if the timebox is exceeded, and if not run it again, while manually incrementing the test count. A better solution would be to implement it in Hedgehog proper, but this hacky solution suited my needs. I'm willing to add it to Hedgehog if there's interest.

From this conversation:

Are you thinking this could be implemented with another combinator?

Yep!

If so, would this be a property level thing? Or could this be supported with a property config option?

Right now I'm leaning in the direction of something on the level of property, i.e. a sibling of checkWith. Some of the qualities of a "TimeBoxedProperty" are at odds with the semantics and types of a normal checkWith test, so I don't think I can use the current PropertyConfig - I'll need a TimeBoxedPropertyConfig. For example, I can imagine a scenario where someone would want to run a test without specifying the testcount and only specifying the timebox. The result will be a test that runs as many times as it can for, say, 1 second. Currently testcount defaults to 100. Therefore it isn't enough to simply add an optional timebox field to PropertyConfig. Size also needs to be optional, and when its in the None case we'll generate random sizes.

Please feel free to consider this idea outside the scope of Hedgehog. I'm pretty sure Haskell doesn't do this!

[TysonMN]

        report() |> loop (i + 1<tests>)
    report() |> loop 1<tests>

Due to F# bug dotnet/fsharp#6984, you should not write tail recursive calls with pipe operators.

[TysonMN]

This all seems reasonable to me. I think it could be directly added to Hedgehog.

[ghost]

For example, I can imagine a scenario where someone would want to run a test without specifying the testcount and only specifying the timebox.

We may need to flesh out some "stop condition" concept in light of this. I've seen this come up in various ways:

• Stop if shrink limit is hit, then fail.
• Stop if test count is hit, then succeed.
• Stop if test runs longer than x, then succeed.

I'd like to hear thoughts on how this could be unified or expressed better. I think keeping the API flexible with combinators and allowing the user to put them together however they wish is the best approach. As you've hinted at, I think we're reaching the limit of the "add another x option" approach.

Edit: Perhaps separating these into two categories would be a good place to start: success conditions, and failure conditions.