New falsify release

Recently Martijn Bastiaan, QBayLogic’s COO, invited Well-Typed to come to QBayLogic HQ and give a one-day workshop on falsify, the new property based testing library that I developed for the Haskell Symposium back in 2023 (paper, presentation). QBayLogic is the company behind Clash: a purely functional language for hardware design. In case you haven’t heard of it, Clash translates Haskell to VHDL or Verilog, which can in turn be translated to actual hardware. It literally uses ghc as its frontend so you have essentially the full power of Haskell available. It’s a great project, I recommend checking it out.

The falsify library takes its main inspiration from the Python Hypothesis library, though it is not a direct translation: it takes the same core idea (“parse, don’t generate”) but reinterprets it in a way that better suits the Haskell way of thinking: more axiomatic approach, support for generating infinite data types (including functions), etc. For more information on falsify, see also the original blog post that announced it, falsify: Hypothesis-inspired shrinking for Haskell.

At QBayLogic HQ we spent the morning developing mini-falsify from scratch, so that we could focus on the main ideas without getting bogged down in the details of the full library; similar in spirit to for example TinyServant, or perhaps Stephen Diehl’s Typechecker Zoo. In the afternoon we hacked on falsify and its application within the clash ecosystem. Partly as a result of that work and partly as a result of me taking this opportunity to do some long overdue maintenance on falsify, there is now a new falsify release: falsify-0.4.0. In the remainder of this blog post we give a brief overview of the main changes.

New feature: Context

Most of this falsify release is just cleanup, but there is one important new feature, spear-headed by Peter Lebbing and Martijn Bastiaan from QBayLogic. The Property monad now has an important new function, called getContext:

getContext :: Property Context

The most important information that the context of a property provides is how many tests we are running for each property (how hard are we trying to falsify this property), and which iteration this particular attempt is.

This can be quite useful, for example when you want to start by looking at small test cases and then slowly broaden the scope. Just as a trivial example, consider the property that “no number is equal to 5”. If we use prim to generate the number to test, producing an arbitrary Word64, the chances that we will find the one counter-example (5) in that enormous search space are essentially non-existent:

demo :: Property ()
demo = do
    x <- gen Gen.prim
    assert $ P.ne .$ ("forbidden", 5)
                  .$ ("x", x :: Word64)

However, we could start with a small range and slowly grow that range; falsify now also offers a convenience function, defined in terms of getContext, for this specific purpose: sized, so-named because it is somewhat similar in spirit to sized in QuickCheck:

sized :: forall e a. (ProperFraction -> a) -> Property a

A ProperFraction is in the half-open interval [0,1); that is, between 0 (inclusive) and 1 (exclusive). We can use this to refine our demo property:

demo2b :: Property ()
demo2b = do
    l <- sized $ ProperFraction.scaleIntegral 100
    x <- gen $ Gen.inRange $ Range.inclusive (0, l)
    assert $ P.ne .$ ("forbidden", 5)
                  .$ ("x", x :: Word64)

This property is easily falsified.

As an aside, I would be somewhat cautious in using this approach. While it is sometimes unavoidable, in general I would recommend generating test case of arbitrary size and then shrinking them down; this is often more likely to actually find counter-examples. If there are specific edge cases that you want to hit, write a generator that covers those edge cases specifically, and perhaps use labelling (label and co) to check that those edge cases are indeed covered. However, as this demo shows, for some properties explicitly searching for small domains first can be very helpful.

Cleanup

The most important change in this release is a cleanup of the code base:

• There is now a separate tasty-falsify package that provides integration with the tasty test framework; falsify itself now provides Test.Falsify.Driver.
• The module hierarchy has been significantly cleaned up. For example, there are now a bunch of new Data.Falsify.* modules for specific datatypes; previously some of these were exported by Test.Falsify.Gen instead; now that module only contains the generators for those datatypes.
• Deprecated functions have been removed
• Some functions have been renamed and some type aliases have been replaced by newtype definitions for increased API clarity.
• All Haddock warnings have been addressed.

For a full list of changes, please refer to the changelog.

Conclusions

Shrinking can be handled manually, in the style of QuickCheck; or automatically, in the style of hedgehog or in the style of falsify. If you are willing to put effort into writing good shrinkers for all your types, then QuickCheck is still your best bet.

If you consider the cost of manually writing shrinkers too large, then within the Haskell ecosystem you have a choice between hedgehog and falsify. The former has the considerable advantage that it’s been around for quite a long time and is a very polished library. The downside is that every time you use monadic bind you introduce a cut-point, which can often result in poor quality shrinking. With Hypothesis showing the way, falsify solves that cut-point problem, though even with falsify it still matters how you write your generators: shrinking is never truly free. Moreover, falsify should be considered an experimental library: it’s nowhere near as battle-tested as Hedgehog, never mind QuickCheck. That said, thanks to the interest of QBayLogic falsify is now a little more mature, so thank you QBayLogic!

For clients who are looking for professional support for the use of falsify, or indeed any other Haskell library, don’t hesitate to contact us at info@well-typed.com.