Functional Programming Unit Testing – Part 3

In the previous post, we talked about using QuickCheck as opposed to traditional xUnit framework testing.  This covered some of the arguments for using property-based testing for our pure algorithms and relegating the xUnit framework tests for the rest.  The question then arises, how do we tie it all together?

But, before we continue, let’s get caught up to where we are today:

 

Integrating QuickCheck and xUnit frameworks

From the previous posts, we’ve talked about running the xUnit framework code and then running our QuickCheck property checks and both using separate frameworks.  My goal is to break down that barrier so that I can have this as part of a continuous integration and be able to run all of my tests at once and have an easy to use mechanism for determining success or failure.

In the Haskell world, how might we do this?

 

Integration HUnit and QuickCheck with Test-Framework

In Haskell, we have two separate frameworks for testing, a traditional xUnit framework in HUnit and a property based testing framework in QuickCheck.   Luckily, there is a framework out there to integrate the two of them together in a rather seamless manner called Test-Framework.  With this tool, we can group our tests together into a list or two, depending on how you want them grouped.  For example, I could take these following four QuickCheck and one HUnit tests together.

Equal
prop_rot13_equals s =
  not (null s) ==> rot13 s == rot13 s

Single is inequal to original
prop_rot13_single_notEquals s =
  not (null s) ==> rot13 s /= s

Double is equal to original          
prop_rot13_double_equals s =  
  not (null s) ==> (rot13 . rot13) s == s

Distribution shapes should be equal  
prop_rot13_group_equals s =
  not (null s) ==> getDistro s == getDistro (rot13 s)
  where getDistro = sort . map length . group . sort

— Empty ROT13 should return empty  
test_rot13_empty =
  assertEqual "rot13 should be empty" [] (rot13 [])
 

And then I could use Test-Framework in order to execute these as a group:

module TestFrameworkTest where

import Test.Framework
import Test.Framework.Providers.HUnit
import Test.Framework.Providers.QuickCheck
import Encryption

tests = 
  [
    testGroup "ROT13 Tests" [
      testProperty "prop_rot13_equals" prop_rot13_equals,
      testProperty "prop_rot13_single_notEquals" prop_rot13_single_notEquals,
      testProperty "prop_rot13_double_equals" prop_rot13_double_equals,
      testProperty "prop_rot13_group_equals" prop_rot13_group_equals,
      testCase "test_rot13_empty" test_rot13_empty]
  ]
    
main = defaultMain tests

 

We can now run our tests from the GHCi window and see that our five tests pass.

ROT13 Tests:
  prop_rot13_equals: [OK, passed 100 tests]
  prop_rot13_single_notEquals: [OK, passed 100 tests]
  prop_rot13_double_equals: [OK, passed 100 tests]
  prop_rot13_group_equals: [OK, passed 100 tests]
  test_rot13_empty: [OK]

         Properties  Test Cases  Total
Passed  4           1           5
Failed  0           0           0
Total   4           1           5

 

As you can see from the above test output, we have a nice way of verifying our results for both our test cases and property-based checks in a cohesive runner.  We can then integrate this into our continuous integration processes for running all of our tests.

In the F# world, what are my options?

 

Integrating FsCheck with Xunit.net

In a post entitled "F# + TestDriven.NET + xUnit.net = WIN", I showed a simple integration with F#, TestDriven.NET and xUnit.net that gives an entire integration story within Visual Studio.  This time, let’s take it a step further to include FsCheck, which is an implementation of QuickCheck 1.0 from the Haskell world.  In order to make this happen, we need to create an IRunner interface instance using an object expression, that I talked about here.

#light

namespace CodeBetter.Samples

module FsCheckExtensions =
  open System
  open FsCheck.Runner
  open Xunit
  
  let xUnitRunner = 
    { new IRunner with
        member x.OnArguments(ntest,args,every)
          args |
          List.iter(function
            | :? IDisposable as d -> d.Dispose() 
            | _ -> ())   
        member x.OnFinished(name, result) =
          match result with
          | True data -> 
              Assert.True(true)
              data.Stamps 
              |> Seq.iter (fun x -> printfn "%d – %A" (fst x) (snd x))
          | False (_,args,None) -> 
              Assert.True(false, sprintf "%s – Falsifiable: %A" name args)
          | False (_,args,Some exc) -> 
              Assert.True(false
                sprintf "%s – Falsifiable: %A with exception %O" name args exc)
          | Exhausted data -> 
              Assert.True(false
                sprintf "Exhausted after %d tests" (data.NumberOfTests) )
        }
  let config = {quick with runner = xUnitRunner}

What this allows me to do is integrate the FsCheck runner with xUnit.net.  As you can see, there are several conditions we must handle such as True, False with or without exception, and Exhausted.  Implementing each is rather simple using the Assert.True function and passing in the given arguments.  I think Exhausted and having an inconclusive state is not proper, so I add that as a failure instead.    Other xUnit based frameworks have the Assert.Fail and Assert.Inconclusive and the latter I don’t like because you should know what you’re testing to know whether it succeeded or not.

Now that our runner is written, we can then integrate our property-based tests into using xUnit.net with this example.  We’ll use our ROT13 implementation again with four tests.  In order to integrate with xUnit.net, we need to call the check function with our given config as we have above, and our property-based test. 

let prop_rot13_equals s =
  not (List.is_empty s) ==> 
    propl (rot13 s = rot13 s)

Then we can define the actual test that xUnit.net will execute using a function decorated with a FactAttribute such as the following:

[<Fact>]
let test_prop_rot13_equals() =  
  check config prop_rot13_equals
 

Running that through the TestDriven.NET runner, we’ll find that our code does indeed pass with flying colors.  In addition, we could add additional tests that don’t use property-based tests to the solution and have them run at the same time such as the following:

[<Fact>]
let rot13_emptyList_should_be_empty() =
  Assert.Empty(rot13 [])
  

Below is the full implementation of the four tests in addition to the startup code required.

#light

namespace CodeBetter.Samples

module EncryptionTests =
  open System 
  open FsCheck
  open FsCheck.Generator
  open Xunit
  // Local imports
  open Encryption
  open ListExtensions
  open FsCheckExtensions

  type CharGenerator() =
    static member Chars = elements([‘A’..‘Z’] @ [‘a’..‘z’])
  
  overwriteGenerators (typeof<CharGenerator>)

  let prop_rot13_equals s =
    not (List.is_empty s) ==> 
      propl (rot13 s = rot13 s)

  [<Fact>]
  let test_prop_rot13_equals() =  
    check config prop_rot13_equals

  let prop_rot13_double_equals s =
    not (List.is_empty s) ==> 
      propl ((rot13 >> rot13) s = s)

  [<Fact>]
  let test_prop_rot13_double_equals() =  
    check config prop_rot13_double_equals
    
  let prop_rot13_single_notEquals s =
    not (List.is_empty s) ==> 
      propl (rot13 s <> s)
  
  [<Fact>]
  let test_prop_rot13_single_notEquals() =  
    check config prop_rot13_single_notEquals
  
  let prop_rot13_group_equals s =
    let getDistro = ListExtensions.defaultSort >> 
                    ListExtensions.group >> 
                    List.map List.length >> 
                    ListExtensions.defaultSort
    not (List.is_empty s) ==> 
      propl (getDistro s = getDistro (rot13 s))
    
  [<Fact>]
  let test_prop_rot13_group_equals() =  
    check config prop_rot13_group_equals
    
  [<Fact>]
  let rot13_emptyList_should_be_empty() =
    Assert.Empty(rot13 [])   

 

When using the runner, we’ll notice that all five tests pass in a relatively short order.  It’s a very nice integration.  You may notice the ListExtensions module that I’m using in my prop_rot13_group_equals function.  Basically, it’s a port of some of the Haskell functions I use often and translated into F#.  Some of these include sort, group, groupBy, span, sort, dropWhile, etc.  I’ll cover the full details in another post, but in the mean time, here are the implementations:

#light

module ListExtensions =
  open System

  let defaultSort (l:#IComparable list)
    l |> List.sort (fun a b -> a.CompareTo(b))

  let rec span (f:‘a -> bool) (l:‘a list) =
    match f, l with
    | _, []             -> [], []
    | p, (x::xs’ as xs) ->
        let ys, zs = span p xs’
        if p x then x::ys, zs else [], xs

  let rec groupBy (f:‘a -> ‘a -> bool) (l:‘a list) =
    match f, l with
    | _,  []    -> []
    | eq, x::xs ->
        let ys, zs = span (eq x) xs
        (x::ys) :: groupBy eq zs

  let group (l:‘a list) = groupBy (=) l

 

Pretty simple, yet powerful additions to the list implementations, in my opinion.  After implementing this bridge between FsCheck and xUnit.net, we can then build this easily into our CI process.

 

Conclusion

In the functional programming world, we have two main ways of testing our code, either through the traditional xUnit tests or the more powerful QuickCheck property-based tests.  Each of these are powerful in their own right, but made more powerful when combined into a single unit.  When combined we have the power of integrating them into our CI process through some of the build/package tools in our tool belt. 

This entry was posted in F#, Functional Programming, Haskell, TDD/BDD. Bookmark the permalink. Follow any comments here with the RSS feed for this post.