I love the idea of FIT tests - customer authored acceptance tests - but the reality seems to fail on both the ability to author and the maintainability of what we create. So every time I return to try again, I always end walking away:

iancooper.spaces.live.com/.../cns!844BD2811F9ABE9C!459.entry

For me it is what Fowler characterizes as symbolic integration. When I make change in the source code it is not reflected in the  FIT tests. Even a simple thing like rename method does not transfer. So what happens is that you have to be code in an API to your source code. Really hurts on the evolutionary design approach.

I (along with several other people I work with) feel the same way about FitNesse.  It's a great first generation tool to get people used to the concept, but we keep looking for the 'new thing' that will replace it.

We've recently decided to start looking into how much duplication there is between our unit tests and our FitNesse tests.  We've are also looking into how much time we spend maintaining the fixture code.  I keep thinking about a new tool that would use reflections to provide a nice interface to the business users that are writing the test cases.  The input and expected output values would be serialized and saved.  If the method signatures change, the tool would display warnings.  However, I think that it would start to require development to be done in a manner that would interact easily with the new tool.  I haven't decided if that's a good thing or not.

We have just started using FITNesse and are using it for acceptance testing/regression testing. The customer is the main user. I haven’t seen any better tool for the customer doing there acceptance testing/regression testing that doesn't require the customer to code. We have used 100% manual acceptance testing up until now and it is not working in the long run, cost are too high.

I agree that FITNesse sucks for thinks that could be tested using xUnit. But xUnit is not something you could give too the customer. If there is no customer involved in the actual testing we tend to use xUnit tests for system tests but using a separate category for those tests so we can exclude them in the continuous integration tests if they depend on values from databases and so on.

If there is work being done on a replacement for FIT/FITNesse that would be even more user friendly and easier to maintain I would love it! Editing the wiki pages are not very user friendly. May be an updated AXAJ based GUI that would use reflection on the backend for intellisense could be a good way forward. For us in the .Net world maybe a nice plug-in for Visual Studio could be an option.

Business Oriented Testing: FIT vs. DSL

My thoughts about why Fit is hard to sell to team

vikasnetdev.blogspot.com/.../why-fit-is-hard-to-sell-to-business.html

I evaluated FIT for a recent project, and ended up writing a "FIT Clone" instead.  Our testing is primarily set based.  Our solution allows the sets to be defined in Excel (no copy and paste to HTML is required).  I'm happy with how it worked out.  The implementation effort was relatively small (just a few days), and we get a lot of flexibility from owning all the source code.  For our requirements, I believe this was a better approach than using FIT.

Pingback from  JEDI  » Blog Archive   » links for 2007-08-13

I've made the same conclusion after our dev team started using Fitnesse last year.  Great approach to testing, horrible tool.  From my observations, a large part of the problem is that Fitnesse is the only game in town for FIT.  And because of that, most non-technical folks don't separate the value of the testing approach from the tool.  

So much of our time is spent managing the tool and making it fit our development environment, and we often forget the whole point of the tool is to save time through automated regression testing.  

Troy,

That was one of the main reasons why we went with our Excel based approach.  The code is compiled right into our app, so it required no change to our deployment process.  Users get to use Excel, which is a tool that they already know and love.  No need to learn a new tool.  No need to deploy any server for Fitnesse.  

Even simple things, like the ability to specify expected results as Excel formulas (based on the inputs) improve usability IMHO.

Hey John, I like the Excel as a testing front end idea. Thanks I'll look at that.

John,

Yeah, we tried using the Data-Driven (excel) Unit tests offered in MSTest (VS2005).  

msdn2.microsoft.com/.../ms182527(VS.80).aspx

We had relatively decent results from this.  It only offers set-based testing like Jeremy discusses--no advanced FIT fixtures, etc.  But in our shop, the overly aggressive Agilists equated Fitnesse with being Agile, so that's where we went.  

Does your Excel solution use the Fit test runner, or do you wire it up to regular unit tests?

Troy,

We do neither of the above :-)   No Fit runner; no unit tests.

The solution is in two halves.  The "front-end" deals with Excel, finding the tables, getting their data, colouring cells to reflect the results.  The "back-end" actually runs those tests against our app.  It consists entirely of custom-written code, specific to our app.  (The dynamic nature of this particular app means we cannot use the usual reflection-based FIT approach, because we do not have a fixed set of fields/properties to reflect on).

I've had good successes writing custom end-to-end test harnesses driven by Excel sheets.

I've used FIT a few times but never FITnesse.  The idea of writing tables in wiki markup and keeping the FIT documents on a wiki server instead of under the same version control system as the rest of the project just gave me the willies.

Hey Jeremy and others, check out Rick Mugridge's new ZiBreve tool, which is also "aimed very squarely as a FitNesse killer for better management and editing of Fit style tests."  ZiBreve, written from scratch in Java, is advertised as a FIT test IDE.

Rick Mugridge, co-author of the book FIT or Developing Software, and author of the entire FitLibrary set of additional FIT fixtures, is the author/vendor for ZiBreve. He is a deeply experienced agile/OO programmer in his own right, a FIT testing fanatic, and an awesome guy. He welcomes input from one and all about how to fix at least some of the issues Jeremy so rightly points out above. ZiBreve is a new tool, not yet completely production-ready, but I have total faith in Rick's committment and ability to get it done, and done right.

Check out ZiBreve here:

www.zibreve.com/visit.php

Cheers,

--Patrick

I am coming to this as a tester who has the unenviable job of trying to validate what the BA's come up with against what the developers produce.  For our pilot project we used the FITLibrary driven from Excel spreadsheets.  The FolderRunner within FITlibrary uses "poi" to interpret the tables held in Excel files.

Spreadsheets are really easy to use ... and guess what, the business created their draft acceptance tests in Excel.  All I had to do was to restructure their tests into simple tables and hey presto within a few days with the developers on board we had FIT tests going green.  

I then did some of the formal testing kind of stuff like class partitioning and some combinatorical analysis to generate a suite of tests scenarios that would make me happy that the system would work.  Doing this flushed out several requirement defects. In particular one bit of workflow was ambiguous.  

The solution was easy.  I sat down with the BA's with a whiteboard and after a couple of hours we had captured what we thought was a viable workflow - and we recorded the results into a spreadsheet.  The business thought it was great and signed it off.  

I did play with FitNesse for a couple of days when I first started looking at FIT.  I then gave up on it because I could see that there was no way I could sell the markup language to the BA's.  I could go as far to say that unless you give BA's an Excel like interface I think its going to be a hard sell.

Another point to take onboard here.  We created over 40 story tests each containing between 5 and 10 subtly different scenarios. From my perspective I only see one fixture that is held in the setup.xls.  With this fixture calling all the re-usable underlying fixtures.  This approach means that after we built the tests for the business rules we could enrich the tests with no need for extensive re-factoring of the tests or the fixture code.

I am still concerned about the re-factoring problem, however I will only have to face it when the business rules or business workflows change.  If I put a QA hat on for a second, then having to revisit all my tests in such a scenario is probably a good thing.

One final thing.  The developers did an initial trial with FIT straight off the download and with the Rick and Ward book in their hands.  They in effect wrote low level FIT tests that had knowledge of the implementation of the system.  This was difficult to extend and maintain, and was unreadable by the business.  This appears to be one of the biggest traps you can fall into in the FIT space.  To my mind I see them as ORTHOGONAL to UNIT tests.  Their purpose is to demonstrate to the BA's and business that the system does what its supposed to do.  My final comment goes to the developers.  "We could have written a test harness using JUnit for this, but it would have taken ages, and it would have still only verified the code quality not that of the requirements"

Oh - by the way - we haven't seen a single defect in this component in system test, UAT or production.

I think some of the pain with Fit comes from the way it commonly ends up being used as a tool for testers to write test scripts out of generic building blocks.

I've created the Concordion framework (Java only, at the moment -sorry) to try and give a stronger emphasis to "illustrating specifications with focused-examples" as opposed to "writing test scripts".

I've found that by making the fixture code simpler developers are happier to write small custom fixtures instead of building a handful of megaliths and this means that the specifications can be written in a more focused and more abstract way.

The drawback with Concordion is that it (currently) requires the specifications to be written in HTML, but (a) it's a pretty easy to learn basic HTML (and that's all you need), and (b) the process works much better if you collaborate. See the website www.concordion.org/Questions.html for more about this.

I worked on and off again for about 2 years on a new OSS project called StoryTeller .  My original

I worked on and off again for about 2 years on a new OSS project called StoryTeller .  My original

I worked on and off again for about 2 years on a new OSS project called StoryTeller .  My original

I've gotten pinged 4-5 times in the past week about just where I'm at with StoryTeller development