Property-based testing (PBT) frameworks come with a number of different features, but which library supports which features? For a PBT newcomer, it can be hard to tell. Strictly speaking you don't even need a PBT framework. Property-based tests can be written from scratch on a case-by-case basis using a random number generator. One can even test stateful code without a state-machine framework, e.g., as outlined here. However a framework provides reusable parts and infrastructure, thus paving the way for bigger developments, such as property-based testing automotive software against the AUTOSAR specification.
- To test an imperative API a framework with state machine support would be nice.
- Integrated shrinking can be a nice feature for bigger developments where writing custom shrinkers may be out of the question.
- ...
This overview is to help myself keep track. It has been compiled over a number of years teaching PBT. As features are gradually added to a framework the table's entries may unfortunately become outdated. YMMV.
I'll be happy to accept PRs for updating entries and adding new frameworks.
| Framework | Language | Gen. EDSL | Shrinking | Int. shr. | State machine | Par. st. mach. | Cov. guidance |
|---|---|---|---|---|---|---|---|
| QuickCheck | Erlang | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | |
| PropEr | Erlang | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | |
| QuickCheck | Haskell | ✔️ | ✔️ | (:heavy_check_mark:) | (:heavy_check_mark:) | ||
| SmallCheck | Haskell | ✔️ | 1 | 1 | |||
| Hedgehog | Haskell | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | |
| Scala-Hedgehog | Scala / JVM | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | |
| R-Hedgehog | R | ✔️ | ✔️ | ✔️ | ✔️ | ||
| FSharp-Hedgehog | F# / .Net | ✔️ | ✔️ | ✔️ | |||
| Hypothesis | Python | ✔️ | ✔️ | ✔️ | ✔️ | (:heavy_check_mark:)2 | (:heavy_check_mark:), (:heavy_check_mark:) |
| TSTL | Python | 3 | ✔️ | ✔️ | ✔️ | ✔️ | |
| ScalaCheck | Scala / JVM | ✔️ | ✔️ | ✔️ | ✔️ | ||
| gopter | Go | ✔️ | ✔️ | ✔️ | |||
| Rapid | Go | ✔️ | ✔️ | ✔️ | ✔️ | ||
| propCheck | Kotlin | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | |
| FsCheck | F# / .Net | ✔️ | ✔️ | ✔️ | |||
| CsCheck | C# / .Net | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | |
| fast-check | JS / TS | ✔️ | ✔️ | ✔️ | ✔️ | (:heavy_check_mark:)2 | |
| QCheck | OCaml | ✔️ | ✔️ | (:heavy_check_mark:) | (:heavy_check_mark:) | (:heavy_check_mark:) | |
| Crowbar | OCaml | ✔️ | (:heavy_check_mark:)4 | ✔️ | |||
| Monolith | OCaml | ✔️ | (:heavy_check_mark:)4 | ✔️ | |||
| Base_quickcheck | OCaml | ✔️ | ✔️ | ||||
| Popper | OCaml | ✔️ | ✔️ | ✔️ | |||
| junit-quickcheck | Java | ✔️ | ✔️ | (:heavy_check_mark:)5 | |||
| QuickTheories | Java | ✔️ | ✔️ | ✔️ | (:heavy_check_mark:) | (:heavy_check_mark:) | ✔️ |
| jqwik | Java | ✔️ | ✔️ | ✔️ | ✔️ | ||
| quickcheck | Rust | ✔️ | ✔️ | ||||
| propcheck | Rust | ✔️ | ✔️ | ✔️ | |||
| Fox | Obj.C / Swift | ✔️ | ✔️ | ✔️ | ✔️ | (:heavy_check_mark:) | |
| SwiftCheck | Swift | ✔️ | ✔️ | ||||
| RapidCheck | C++ | ✔️ | ✔️ | ✔️ | (:heavy_check_mark:) | ||
| test.check | Clojure | ✔️ | ✔️ | ✔️ | ? | ? | |
| Lua-QuickCheck | Lua | ✔️ | ✔️ | ✔️ | ✔️ | ||
| theft | C | (:heavy_check_mark:) | ✔️ | (:heavy_check_mark:) | (:heavy_check_mark:) | ||
| DeepState | C / C++ | (:heavy_check_mark:) | ✔️ | ✔️ | ✔️ | ||
| Echidna | Solidity / EVM | ✔️ | ✔️ | ✔️ | ✔️ | ||
| ... |
Legend:
- Language denotes the frameworks's language or platform
- Generator EDSL denotes whether the framework has an expressive, embedded domain-specific language for programming custom generators (
int,list,map, ...) - Shrinking denotes whether the framework has built-in support for reducing counterexamples.
- Integrated shrinking denotes whether a shrinker automatically preserves any invariants of an EDSL-based custom generator (sorted lists, non-empty array, valid JSON, ...)
- State machine denotes whether the framework has a state-machine library for model-based testing.
- Parallel state machine denotes whether the framework supports parallel state-machine testing for race conditons, etc.
- Coverage guidance denotes whether the framework's generators (and shrinkers) can be guided by code coverage information obtained via instrumentation.
Footnotes
- 1 Contrary to QuickCheck and its descendants, SmallCheck's generators enumerate values starting from the smallest ones (up to some bound). SmallCheck will therefore encounter a minimal counterexample first and hence doesn't require shrinking.
- 2 Hypothesis and fast-check support asynchronous state machine testing, which can find race conditions (although it is strictly speaking not using parallel testing).
- 3 TSTL instead uses an external DSL.
- 4 Crowbar and Monolith both use AFL which trims each test input as part of its core genetic algorithm. In addition they support test case reduction via
afl-tminwhich is unaware of (and hence may break) OCaml typing. - 5 JQF and the underlying Zest supports multiple forms feedback guidance beyond coverage.
The term property-based testing seems to originate from 'Property-Based Testing; A New Approach to Testing for Assurance' by Fink and Bishop (SE Notes 1997). The approach was popularized as an embedded domain-specific language in 'QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs' by Claessen and Hughes (ICFP 2000) which inspired ports to many other languages.
Integrated shrinking is explained in more detail in
- an early design thread on the Haskell mailing list
- a Reddit thread announcing Hedgehog
- a talk by Jacob Stanley on Hedgehog's design
- a blog post by Edsko de Vries with pro/cons
- McIver's Hypothesis article advocating the approach
- 'Test-Case Reduction via Test-Case Generation: Insights From the Hypothesis Reducer' by MacIver and Donaldson (ECOOP 2020)
- Everything You Ever Wanted to Know About Test-Case Reduction, But Didn’t Know to Ask - describing DeepState's reducer/shrinker
State machines to test protocols and systems with state are described in
- 'Testing reactive systems with GAST' by Koopman and Plasmeijer (TFP 2003, revised 2005) - which describes a model-based framework for Clean's Gast library
- 'Testing Telecoms Software with Quviq QuickCheck' Arts, Hughes, and Johansson (Erlang 2006) - which describes Erlang's first
eqc_commandsmodule - 'QuickCheck Testing for Fun and Profit' by Hughes (PADL 2007) describes a later revision
- the API documentation of Quviq's latest (commercial) version
- 'A Note on State-Machine Frameworks for Property-Based Testing' - a tutorial which reconstructs
qcstmfor OCaml
Parallel state-machine tests for race conditions were later introduced in
- 'Finding Race Conditions in Erlang with QuickCheck and PULSE' by Claessen et al. (ICFP 2009) -- along with a scheduler and a process visualizer
- 'Testing a Database for Race Conditions with QuickCheck' by Hughes and Bolinder (Erlang 2011)