Sponsored By Aspose - File Format APIs for .NET

Aspose are the market leader of .NET APIs for file business formats – natively work with DOCX, XLSX, PPT, PDF, MSG, MPP, images formats and many more!

Why I’m suddenly down on Fit/FitNesse

Quick Note:  I love *what* Fit testing accomplishes, but the mechanics are an issue.  There’s a rumor floating around that a bunch of test automation gurus are going to work up some ideas for how to “save” Fit testing.  Anybody feel like commenting?  I don’t know anymore details that that.

I’ve had doubts for months, but I think I finally soured on Fit testing within about 5 minutes last week.  Here’s why:

  1. To fix an existing test I caught myself using 2-3 different low level tricks within Fit.  I built a couple different custom type handler and injected them, overrode a DoTables() method, and some other bit of hackery that involved low level knowledge of Fit internals.  I found myself using 4-5 different Fixture classes in a single class.  It suddenly occurred to me that there was no way in hell that I was going to be able to completely transition this stuff to the client’s testers and developers before I leave (especially since I work beside them instead of working with them, but that’s a different battle).  It just requires too much internal knowledge of Fit mechanics to do anything outside of basic set-based testing.
  2. Nat Pryce aptly described the Fit engine mechanics as “reflectopornographic“, and I don’t want to use something described as “reflectopornographic.”  If you’ve worked with Fit testing before, you’re probably laughing along with that right now.
  3. Some things are easier in Fit tests than xUnit tests, but for larger flow based tests I’d rather just write code alone.  If I had it to do over again, I’d keep Fit (StoryTeller in reality) for set-based things, but would have written a, wait for it, fluent interface test harness for the flow based testing I do against the user interface.  I don’t have any participation from the customer and little from the tester, so why not use the tools I’m most comfortable with?  I’m spending too much time hunting down test grammars from the Fixture classes to remember what to type into the Fit editing window.  Intellisense anyone?

When I say set-based here, I’m talking about tests that can very accurately be expressed completely in 2 dimensional tables.  X number of inputs leading to Y number of outputs.  Fit excels at that with the ColumnFixture and RowFixture tests.

 

What about StoryTeller?

I’ve just done a reset on the StoryTeller UI and I’m committed to finishing it.  StoryTeller is aimed very squarely as a FitNesse killer for better management and editing of Fit style tests, not as a replacement of the actual FitnesseDotNet engine.  I think in the longer term I’m keeping StoryTeller for the things it is good at, but start moving more towards the xBehave tools for the flow style testing that I use DoFixture’s for today.

 

Regardless of whether or not you’re using Fit happily, with reservations, or just want to think about starting, I’m putting up a lessons learned post in the next couple days on Fit testing in specific and automated acceptance tests in general.

About Jeremy Miller

Jeremy is the Chief Software Architect at Dovetail Software, the coolest ISV in Austin. Jeremy began his IT career writing "Shadow IT" applications to automate his engineering documentation, then wandered into software development because it looked like more fun. Jeremy is the author of the open source StructureMap tool for Dependency Injection with .Net, StoryTeller for supercharged acceptance testing in .Net, and one of the principal developers behind FubuMVC. Jeremy's thoughts on all things software can be found at The Shade Tree Developer at http://codebetter.com/jeremymiller.
This entry was posted in StoryTeller. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Rob Whitener

    I have been developing tests for FitNesse for about 4 years now. And when I say developing tests, I mean the full stack. From putting the tables into the wiki all the way down to the application glue layer. Now, I am tired, so very tired.

    I had very high hopes for FitNesse – I thought the goals were concise and easy to understand whether you were a developer, tester, or BA. Build the right code, build it correctly. You know you are done when the tests run green. Unfortunately, like golf, even though the objective is supremely simple, actually getting there is almost impossibly difficult.

    We began a project, a big project, as a waterfall team trying to go Agile. I was still rather fresh out of CS classes and still rather naive about how shit really works. I thought I would be able to convince developers to write JUnit tests and FitFixtures with me, I would provide app hooks, and then the manual testers would be the ones creating the tables. After 4 years of trying, it never happened. And believe me, I tried. I am now very comfortable standing up and giving a presentation to a bunch of super critical people who aren’t really interested in what I have to say. That is probably the best thing that came out of the experience, since no one else but me has ever written any fit tests.

    People did try though. The wiki interface is far to clunky for people to use effectively to create test cases. It works pretty good as a plain old wiki (actually, the place it got the most use outside of my efforts was as a documentation hub!). As far as the development team was concerned. They were mostly used to developing in COBOL on a mainframe. They had to learn Java in order to complete the project – but expecting them to also write automated tests prior to writing the code was just a bridge too far.

    I have managed to cobble together a pretty healthy set of automated tests over the years, and I believe it has provided real value in being able to quickly (in relative terms) turn around a regression test. However, the tests have become difficult to maintain and I have grown tired of fighting with them. Part of the problem also rests with the system. We are a B2B e-commerce provider, we provide EDI based trading relationships to our customers. So, most of our functional testing is system integration testing involving multiple communctionat protocols (FTP, HTTP, proprietary protocols). When I send a message to a test user, the delay on the test system could be 1 second or 1 minute. The web based portion was written by people who have never built a web app before, and the core of the application was built on the webMethods platform which has its own set of issues. But I digress.

    Driving adoption of a tool like FitNesse is difficult. None of the examples in the fit book represent software systems that are very large in scope, leading me to believe that tools like Fit are really only valuable to small projects with a singular focus and not too much external service integration. If you work in a small web app shop, they are great. However, should you be trying to test enterprise software, unless you have software engineering skills on your testing team, Fit will probably fail you.

    In addition to falling out of love with FitNesse, I also fell out of love with Java – I am now in a lusty wildfire type relationship with Ruby and in fact, I have been writing my own FitNesse killer in Ruby! (On the current project, my last major test development was all ruby, and man, it was a joy to develop).

    I have also built a tool that I have put on RubyForge called ActiveExcel, which acts like an ORM for Excel. You can treat rows in an excel spreadsheet like objects in ruby. pretty neat, but still a tiny bit buggy. If you want to try it out: gem install ActiveExcel

    Using active excel you can represent test cases in Excel (easy to create alot of test fast) and then build your test harness in Ruby (once you get the hang of it, its pretty easy to code in). These won’t be story tests, by any stretch of the imagination. But, this type of testing lends itself well to areas where you need to test out alot of different inputs (boundary testing, negative testing, etc.)

    Happy Testing!

  • http://www.concordion.org/ David Peterson

    I think some of the pain with Fit comes from the way it commonly ends up being used as a tool for testers to write test scripts out of generic building blocks.

    I’ve created the Concordion framework (Java only, at the moment -sorry) to try and give a stronger emphasis to “illustrating specifications with focused-examples” as opposed to “writing test scripts”.

    I’ve found that by making the fixture code simpler developers are happier to write small custom fixtures instead of building a handful of megaliths and this means that the specifications can be written in a more focused and more abstract way.

    The drawback with Concordion is that it (currently) requires the specifications to be written in HTML, but (a) it’s a pretty easy to learn basic HTML (and that’s all you need), and (b) the process works much better if you collaborate. See the website http://www.concordion.org/Questions.html for more about this.

  • http://martin.p.jackson@jpmchase.com Martin P Jackson

    I am coming to this as a tester who has the unenviable job of trying to validate what the BA’s come up with against what the developers produce. For our pilot project we used the FITLibrary driven from Excel spreadsheets. The FolderRunner within FITlibrary uses “poi” to interpret the tables held in Excel files.

    Spreadsheets are really easy to use … and guess what, the business created their draft acceptance tests in Excel. All I had to do was to restructure their tests into simple tables and hey presto within a few days with the developers on board we had FIT tests going green.

    I then did some of the formal testing kind of stuff like class partitioning and some combinatorical analysis to generate a suite of tests scenarios that would make me happy that the system would work. Doing this flushed out several requirement defects. In particular one bit of workflow was ambiguous.

    The solution was easy. I sat down with the BA’s with a whiteboard and after a couple of hours we had captured what we thought was a viable workflow – and we recorded the results into a spreadsheet. The business thought it was great and signed it off.

    I did play with FitNesse for a couple of days when I first started looking at FIT. I then gave up on it because I could see that there was no way I could sell the markup language to the BA’s. I could go as far to say that unless you give BA’s an Excel like interface I think its going to be a hard sell.

    Another point to take onboard here. We created over 40 story tests each containing between 5 and 10 subtly different scenarios. From my perspective I only see one fixture that is held in the setup.xls. With this fixture calling all the re-usable underlying fixtures. This approach means that after we built the tests for the business rules we could enrich the tests with no need for extensive re-factoring of the tests or the fixture code.

    I am still concerned about the re-factoring problem, however I will only have to face it when the business rules or business workflows change. If I put a QA hat on for a second, then having to revisit all my tests in such a scenario is probably a good thing.

    One final thing. The developers did an initial trial with FIT straight off the download and with the Rick and Ward book in their hands. They in effect wrote low level FIT tests that had knowledge of the implementation of the system. This was difficult to extend and maintain, and was unreadable by the business. This appears to be one of the biggest traps you can fall into in the FIT space. To my mind I see them as ORTHOGONAL to UNIT tests. Their purpose is to demonstrate to the BA’s and business that the system does what its supposed to do. My final comment goes to the developers. “We could have written a test harness using JUnit for this, but it would have taken ages, and it would have still only verified the code quality not that of the requirements”

    Oh – by the way – we haven’t seen a single defect in this component in system test, UAT or production.

  • http://codebetter.com/blogs/jeremy.miller Jeremy D. Miller

    Actually Patrick, I’m well on my way to a .Net flavored FitNesse killer –> http://storyteller.tigris.org

  • Patrick Wilson-Welsh

    Hey Jeremy and others, check out Rick Mugridge’s new ZiBreve tool, which is also “aimed very squarely as a FitNesse killer for better management and editing of Fit style tests.” ZiBreve, written from scratch in Java, is advertised as a FIT test IDE.

    Rick Mugridge, co-author of the book FIT or Developing Software, and author of the entire FitLibrary set of additional FIT fixtures, is the author/vendor for ZiBreve. He is a deeply experienced agile/OO programmer in his own right, a FIT testing fanatic, and an awesome guy. He welcomes input from one and all about how to fix at least some of the issues Jeremy so rightly points out above. ZiBreve is a new tool, not yet completely production-ready, but I have total faith in Rick’s committment and ability to get it done, and done right.

    Check out ZiBreve here:

    http://www.zibreve.com/visit.php?page=index

    Cheers,

    –Patrick

  • Nat

    I’ve had good successes writing custom end-to-end test harnesses driven by Excel sheets.

    I’ve used FIT a few times but never FITnesse. The idea of writing tables in wiki markup and keeping the FIT documents on a wiki server instead of under the same version control system as the rest of the project just gave me the willies.

  • John Rusk

    Troy,

    We do neither of the above :-) No Fit runner; no unit tests.

    The solution is in two halves. The “front-end” deals with Excel, finding the tables, getting their data, colouring cells to reflect the results. The “back-end” actually runs those tests against our app. It consists entirely of custom-written code, specific to our app. (The dynamic nature of this particular app means we cannot use the usual reflection-based FIT approach, because we do not have a fixed set of fields/properties to reflect on).

  • Troy Tuttle

    John,
    Yeah, we tried using the Data-Driven (excel) Unit tests offered in MSTest (VS2005).

    http://msdn2.microsoft.com/en-us/library/ms182527(VS.80).aspx

    We had relatively decent results from this. It only offers set-based testing like Jeremy discusses–no advanced FIT fixtures, etc. But in our shop, the overly aggressive Agilists equated Fitnesse with being Agile, so that’s where we went.

    Does your Excel solution use the Fit test runner, or do you wire it up to regular unit tests?

  • http://iancooper.spaces.live.com/ Ian Cooper

    Hey John, I like the Excel as a testing front end idea. Thanks I’ll look at that.

  • http://www.agilekiwi.com John Rusk

    Troy,

    That was one of the main reasons why we went with our Excel based approach. The code is compiled right into our app, so it required no change to our deployment process. Users get to use Excel, which is a tool that they already know and love. No need to learn a new tool. No need to deploy any server for Fitnesse.

    Even simple things, like the ability to specify expected results as Excel formulas (based on the inputs) improve usability IMHO.

  • Troy Tuttle

    I’ve made the same conclusion after our dev team started using Fitnesse last year. Great approach to testing, horrible tool. From my observations, a large part of the problem is that Fitnesse is the only game in town for FIT. And because of that, most non-technical folks don’t separate the value of the testing approach from the tool.

    So much of our time is spent managing the tool and making it fit our development environment, and we often forget the whole point of the tool is to save time through automated regression testing.

  • http://www.activesharp.com John Rusk

    I evaluated FIT for a recent project, and ended up writing a “FIT Clone” instead. Our testing is primarily set based. Our solution allows the sets to be defined in Excel (no copy and paste to HTML is required). I’m happy with how it worked out. The implementation effort was relatively small (just a few days), and we get a lot of flexibility from owning all the source code. For our requirements, I believe this was a better approach than using FIT.

  • http://vikasnetdev.blogspot.com/ Vikas Kerni
  • http://hakanforss.spaces.live.com Håkan Forss

    We have just started using FITNesse and are using it for acceptance testing/regression testing. The customer is the main user. I haven’t seen any better tool for the customer doing there acceptance testing/regression testing that doesn’t require the customer to code. We have used 100% manual acceptance testing up until now and it is not working in the long run, cost are too high.
    I agree that FITNesse sucks for thinks that could be tested using xUnit. But xUnit is not something you could give too the customer. If there is no customer involved in the actual testing we tend to use xUnit tests for system tests but using a separate category for those tests so we can exclude them in the continuous integration tests if they depend on values from databases and so on.
    If there is work being done on a replacement for FIT/FITNesse that would be even more user friendly and easier to maintain I would love it! Editing the wiki pages are not very user friendly. May be an updated AXAJ based GUI that would use reflection on the backend for intellisense could be a good way forward. For us in the .Net world maybe a nice plug-in for Visual Studio could be an option.

  • http://www.scottlilly.com Scott Lilly

    I (along with several other people I work with) feel the same way about FitNesse. It’s a great first generation tool to get people used to the concept, but we keep looking for the ‘new thing’ that will replace it.

    We’ve recently decided to start looking into how much duplication there is between our unit tests and our FitNesse tests. We’ve are also looking into how much time we spend maintaining the fixture code. I keep thinking about a new tool that would use reflections to provide a nice interface to the business users that are writing the test cases. The input and expected output values would be serialized and saved. If the method signatures change, the tool would display warnings. However, I think that it would start to require development to be done in a manner that would interact easily with the new tool. I haven’t decided if that’s a good thing or not.

  • Joesph Gutierrez

    For me it is what Fowler characterizes as symbolic integration. When I make change in the source code it is not reflected in the FIT tests. Even a simple thing like rename method does not transfer. So what happens is that you have to be code in an API to your source code. Really hurts on the evolutionary design approach.

  • http://iancooper.spaces.live.com Ian Cooper

    I love the idea of FIT tests – customer authored acceptance tests – but the reality seems to fail on both the ability to author and the maintainability of what we create. So every time I return to try again, I always end walking away:

    http://iancooper.spaces.live.com/blog/cns!844BD2811F9ABE9C!459.entry