How to integrate FitNesse acceptance tests into your CC.Net build – level 300

I’m doing a lot with FitNesse lately, and it’s going quite well.  Unit testing ensures that each component is doing what it is supposed to, and integration testing ensures that the components work well together.  Acceptance tests are a bit different.  They actually test that the software does what the customers think it’s doing.  They test that the developer understood the business need.  They give the customer (or product manager) assurance that the system actually works.

With FitNesse, our product manager (and crew) can  use a simple wiki to exercise the system.  The developers create test Fixtures that speak in the domain language but actually exercise the system under test.  A user may type:

!|fit.ActionFixture|
|start|Emailer|
|press|SendDefaultEmail|
|check|NumEmailsSent|

This simple table is enough to actually exercise the system under test and get it to do something.  If all is well, the Emailer will send a default email.  If this test fails, it saves the company from a black eye when a user encounters the problem.

It was a little tricky to get the FitNesse test integrated with our CC.Net build.  We wanted the acceptance test to be a current status report of where we are.  When all the acceptance tests are passing, we are done.  Consequently, we didn’t want failing acceptance tests to break the build.  Only NUnit tests break the build because they should _always_ be passing.  No problem.  We create a new target with some exec tasks:

<exec program=”<path to TestRunner.exe>” commandline=”-results <some directory>\FitNesse-Results.html <FitNesse server> <port> <TestSuite>” failonerror=”false”/>

This will actually call the FitNesse server (with the wiki on it) and execute all the acceptance tests.  This will produce a file in the raw FitNesse format.  We’d like it in xml so we can incorporate it in the CC.Net build report.  Xml transforming isn’t a part of the .Net port, but it is in the Java version, so we’ll just use it.

<exec program=”java.exe” commandline=”-cp bin\fit\fitnesse.jar fitnesse.runner.FormattingOption <some directory>\FitNesse-Results.html xml <some directory>\Fit-Results.xml <FitNesse server> <port> <TestSuite>” failonerror=”false”/>

This hooks into the transform logic on the FitNesse server to change our raw output file to a nice Xml format.  Now we have something that CruiseControl.Net can use for a build report and email.  We will have to make an Xsl, though.  Here’s a simple Xsl that will pull out the FitNesse summary information and through it out to your CC.Net build summary:

<?xml version=”1.0″?>

<xsl:stylesheet

xmlns:xsl=”http://www.w3.org/1999/XSL/Transform” version=”1.0″>

<xsl:output method=”html”/>

<xsl:variable name=”fit.result.list” select=”//testResults/result”/>

<xsl:variable name=”fit.wrongpagecount” select=”countU$fit.result.list/counts/wrong)” />

<xsl:variable name=”fit.ignorespagecount” select=”countU$fit.result.list/counts/ignores)” />

<xsl:variable name=”fit.exceptionspagecount” select=”countU$fit.result.list/counts/exceptions)” />

<xsl:variable name=”fit.correctpagecount” select=”countU$fit.result.list/counts[wrong/text() = 0 and exceptions/text() = 0 and ignores/text() = 0])” />

<xsl:variable name=”fit.correctcount” select=”//testResults/finalCounts/right”/>

<xsl:variable name=”fit.failures” select=”//testResults/finalCounts/wrong”/>

<xsl:variable name=”fit.notrun” select=”//testResults/finalCounts/ignores”/>

<xsl:variable name=”fit.exceptions” select=”//testResults/finalCounts/exceptions”/>

<xsl:variable name=”fit.case.list” select=”$fit.result.list//test-case”/>

<xsl:variable name=”fit.suite.list” select=”$fit.result.list//test-suite”/>

<xsl:variable name=”fit.failure.list” select=”$fit.case.list//failure”/>

<xsl:variable name=”fit.notrun.list” select=”$fit.case.list//reason”/>

<xsl:variable name=”colorClass”>

<xsl:choose>

<xsl:when test=”$fit.exceptionspagecount > 0″>fiterror</xsl:when>

<xsl:when test=”$fit.ignorespagecount > 0″>fitignore</xsl:when>

<xsl:when test=”$fit.wrongpagecount > 0″ >fitfail</xsl:when>

<xsl:otherwise>fitpass</xsl:otherwise>

</xsl:choose>

</xsl:variable>

<xsl:variable name=”fit.tests.present” select=”countU//testResults/result) > 0 or count(/cruisecontrol/build/buildresults//testsuite) > 0″ />

<xsl:template match=”/”>

<xsl:choose>

<xsl:when test=”$fit.tests.present”>

<style>

*.fitpass{

background-color: #AAFFAA;

}

*.fitfail{

background-color: #FFAAAA;

}

*.fiterror

{

background-color: #FFFFAA;

}

*.fitignore

{

background-color: #CCCCCC;

}

*.fitheader{

border: solid 1px black;

margin: 1px;

padding: 2px;

}

*.line{

margin: 5px;

}

</style>

<div>

<div class=”{$colorClass} fitheader”>

<strong>FitNesse Summary — Test Pages:</strong> <xsl:value-of select=”$fit.correctpagecount”/> right, <xsl:value-of select=”$fit.wrongpagecount”/> wrong,

<xsl:value-of select=”$fit.ignorespagecount”/> ignored, <xsl:value-of select=”$fit.exceptionspagecount”/> exceptions

<strong>Assertions:</strong> <xsl:value-of select=”$fit.correctcount”/> right, <xsl:value-of select=”$fit.failures”/> wrong,

<xsl:value-of select=”$fit.notrun”/> ignored, <xsl:value-of select=”$fit.exceptions”/> exceptions

</div>

<xsl:for-each select=”$fit.result.list”>

<xsl:variable name=”colorClass”>

<xsl:choose>

<xsl:when test=”counts/exceptions > 0″>fiterror</xsl:when>

<xsl:when test=”counts/ignores > 0″>fitignore</xsl:when>

<xsl:when test=”counts/wrong > 0″ >fitfail</xsl:when>

<xsl:otherwise>fitpass</xsl:otherwise>

</xsl:choose>

</xsl:variable>

<div class=”line”>

<span class=”{$colorClass}” style=”padding:2px;” ><xsl:value-of select=”counts/right”/> right, <xsl:value-of select=”counts/wrong”/> wrong,

<xsl:value-of select=”counts/ignores”/> ignored, <xsl:value-of select=”counts/exceptions”/> exceptions </span>

<span style=”padding:2px;”><xsl:value-of select=”relativePageName”/></span>

</div>

</xsl:for-each>

</div>

</xsl:when>

</xsl:choose>

</xsl:template>

</xsl:stylesheet>

 

Hook this Xsl into your CC.Net configuration, and that’s as hard as it is!  FitNesse results “magically” show up in the build summary.  It’s really a big win for visibility into the state of the software.  The build summary is now the actual status report of the software, and it’s not subjective.  What does 83% actually mean?  With a list of acceptance tests to fulfill, we know exactly when we are done.