Most approaches to testing test individual examples. This is often not enough to tease out rare but realistic edge cases. As a result, it takes a large number of individual tests to really be confident that our API is ready for deployment: This is a lot of work. What if we could generate test cases automatically?

Doing so leads to thinking about tests not in terms of individual examples, but general properties, and from there to an approach called property-based testing or QuickCheck. QuickCheck generates tests from properties. It has its origins in functional programming and has been a crucial tool in eliminating bugs from many complex software projects. But QuickCheck does more than help find bugs: It encourages thinking about our API in terms of properties, and this style of thinking often leads to interesting domain insights and architectural improvements.

APIs and Specifications

Consider the following API for a simple “smart” shopping cart, written in F#:

type Interface =
  abstract member Add: int -> Item -> unit
  abstract member Total: unit -> Price

The first method, Add, takes an integer representing a count, and an item to be added to the cart, and returns nothing. (unit is used in F# similarly to void in other languages.) The second method returns a total price for all items in the cart. (That’s why it’s “smart” – it knows the prices of the items in it.)

We add a convenience module with functions for calling the methods:

module Interface =
let add (cart: Interface) (count: int) (item: Item) = cart.Add count item
let total (cart: Interface): Price = cart.Total ()

We also add a factory interface, creating a shopping-cart implementation from a catalog assigning a price to each item:

type Catalog = Map<Item, Price>

type Factory =
abstract member make: Catalog -> Interface

Now, the types give us some information about what kinds of values are input and output to the various operations, but not what those values actually are. How could we obtain that information? We could look at the implementation, but that might be messy and complex – databases and microservices might be involved. We could look at unit tests, which common development methodologies produce as a matter of course. These might look as follows, assuming there is a factory for shopping carts available as factory:

let test =
let cart1 = factory.make Map.empty
assertEquals (Interface.total cart1) (Price 0)
let cart2 = factory.make (Map.ofList [Item "Milk", Price 100; Item "Egg", Price 20])
assertEquals (Interface.total cart2) (Price 0)

(We assume a function assertEquals from some test frameworks, and that items and prices are constructed with the Item and Price constructors.)

These are trivial tests, but they do make a point: The first one says that the total a freshly created shopping cart is 0 – provided its catalog is empty. The second one says the same thing for a catalog containing only milk and eggs. The TDD community calls this an “executable specification” (or part of one), but there is still a lot missing. Looking at it, you might say “The total is 0 for all shopping carts, for all possible catalogs.” This is in fact a property of the shopping-cart factory, and, even though may seem trivial, worth stating explicitly. This could look as follows:

let total0Correct =
Prop.forAll Arb.catalog (fun catalog ->
let cart = factory.make catalog
Interface.total cart .=. Price 0)

The Prop and Arb modules come from the FsCheck package. Prop.forAll returns a property (of type Property) that should hold for all values from a certain set. Arb.catalog means “arbitrary catalog”, and Prop.forAll accepts a function for naming that catalog. The body of that function is a boolean expression that first creates a cart associated with the catalog, and then checks whether the total reported by the cart is indeed 0. As opposed to the two “example unit tests” above, this states that the total is 0 for all catalogs.

FsCheck provides a function called quick that checks whether the property holds. In F# Interative, it prints output as follows:

quick total0Correct;;
Ok, passed 100 tests.

FsCheck has just generated 100 tests from the description of the property. We can instrument total0Correct to print out the catalogs of those tests:

map [(Item "HF", Price 1); (Item "dK", Price 2)]
map [(Item "Bx", Price 1); (Item "QaY", Price 1); (Item "ca", Price 3)]
map [(Item "Qhwu", Price 2); (Item "t", Price 3)]
map [(Item "HS", Price 1); (Item "h", Price 3); (Item "uj", Price 1)]
map [(Item "IXVgqo", Price 1); (Item "Rgr", Price 5); (Item "Zgnh", Price 3)]
map [(Item "ZXhwG", Price 4); (Item "nwFUSd", Price 2)]

This means that FsCheck can generate many tests from a declarative description of a property. FsCheck’s underlying idea comes from one of the most influential works in functional programming...