Discovering the value of QuickCheck

I like tests. I like the confidence that you get when you have an extensive test suite so you know you'll be notified when things break. But so far I've used mainly frameworks like HUnit to explicitly state my assertions. I know what the code should produce given the inputs, and I encode that information in tests. So I didn't really see what QuickCheck brought to that. I'm still not that comfortable with the random aspect of it, but I understand that it can find corner cases for you, whereas with assertions the corner cases you need to think of beforehand, or add them when your users encounter them (ouch). But I'm starting to see the value of QuickCheck.

I was working inside the HTF source code, and the goal was to produce a textual representation of a diff between strings similar to what the diff utility does on Unix. I just wanted to have a fallback for machines that don't have diff installed. And Stefan had written a QuickCheck property to verify that the outputs of the pure Haskell code and the utility matched. I used the property to fall back to my comfortable way of working: fire QuickCheck, let it find a failing case, add it to a HUnit TestCase, and try to understand and fix the issue without breaking the rest. So QuickCheck was just a test case generator.

But then Sterling, the maintainer of Diff, convinced me that 100% matches between the code and the utility was an utopian goal, since diff does some tradeoffs for speed and compactness, and rewriting diff with all its quirks wasn't really worthwhile. And that's when I started to like QuickCheck. I thought: we want exact matches most of the time, but it's fine to have differences as long as what we generate is not too big compared to what the diff utility outputs. This is what QuickCheck classify function can do: I first verify if the size of the output is ok (less than 10% bigger that the utility output), and then I classify the test case as an exact match if the two outputs match. And then I use the cover function (not, ahem, covered in the manual ) to ensure that I get at least  90% exact match:


cover (haskDiff == utilDiff) 90 "exact match" $ 
  classify (haskDiff == utilDiff) "exact match"    
    (div ((length haskDiff)*100) (length utilDiff) < 110)


So the tests are random, yes, but I've pretty good confidence that the output function does what's it's supposed to do. 90% of the time :-).