From a TDD Fanboy’s viewpoint, how you know your approach to TDD or just plain unit testing is working?

I’ve had a couple conversations the past couple weeks about extending test coverage in the user interface and the necessity of writing integrated tests.  One person wished his team were doing more automated testing through the UI (web app), and the other person just assumed that testing the UI (WinForms) itself was impossible.  For the sake of this conversation, let’s say that it is perfectly possible to test the actual user interface.  That leaves us with the question “is it feasible or even necessary to write automated tests against the UI?”

Let’s broaden the question out to “how do I know my unit and integration testing is good enough?”  Here’s my simple, but subjective, criteria to knowing if the developer testing approach is sufficient:

  • Are you getting defects that seem easily preventable?  Oddball combinations of user interactions and edge cases don’t count.  If you’re getting a perfectly manageable rate of defects from the testers, then I’d say you’re doing ok.  Unfortunately, my experience is that most of the bugs are related to screen behavior.  I actually write quite a few unit tests purely on screen to presenter wiring.  Going farther, my particular application contains so much screen behavior that I also invest in writing an extensive battery of integration tests between the MVP triad with the rest of the application services stubbed out just to get the interaction of the MVP pieces down.  It’s turned up a lot of problems and helped stop regression bugs.  That investment wouldn’t pay itself off on many projects, but it does on mine.
  • Are you having to use the debugger quite a bit on your own code?  This might be the key indicator of the effectiveness of your unit testing.  If you’re still having to use the debugger to any extent, my first thought would be to start making your unit testing more granular.  Good unit tests/specifications keep that debugger collecting dust.  Since I’d argue that the short term value proposition of doing TDD is to move more time out from the debugging column than you have to move into the unit testing column.  If you’re still debugging, then TDD isn’t paying off.  I learned this lesson in a hard way when I was doing some of the earlier work on StructureMap.  In later releases I made changes to the architecture just to create more seams for easier unit testing.  Those seams made the feedback loop faster and finer grained so I just really didn’t need my debugger quite so much.

 

From a comment on programming.reddit.com:

“The TDD fanboys are swarming to defend their dogma. From a comment, “If TDD hurts, it’s because you’re doing it wrong” (!!!).”

Sarcasm aside, this quote is dead on accurate.  Pain is a stimuli.  Pain tells us to stop doing whatever it is we’re doing for our own good.  If your arm hurts when you bend it that way, stop bending it that way!  If doing TDD on your project hurts, then you’re definitely doing something that isn’t right for your project.

I have yet to work in a problem domain where I couldn’t effectively use TDD for most of the code I wrote, but I’m sure that that domain is out there.  On the other hand, to make TDD go smoothly you often have to change the way you construct the code and structure responsibilities to take advantage of TDD.  In many cases that change coincides with what I would call good design anyway.  In my experience, code that makes tests difficult or laborious to setup is almost always suffering from tight coupling and low cohesion — problems that you really don’t want regardless of using TDD.  If you have questioned your design, but still can’t get to a point where writing and maintaining the unit tests first is possible, then yeah, ditch TDD because it’s doing more harm than good.   TDD is merely a means.  Working software is the goal.  If you’re happy with your results, all’s well.  Unless your competitors are continuing to get better…

About Jeremy Miller

Jeremy is the Chief Software Architect at Dovetail Software, the coolest ISV in Austin. Jeremy began his IT career writing "Shadow IT" applications to automate his engineering documentation, then wandered into software development because it looked like more fun. Jeremy is the author of the open source StructureMap tool for Dependency Injection with .Net, StoryTeller for supercharged acceptance testing in .Net, and one of the principal developers behind FubuMVC. Jeremy's thoughts on all things software can be found at The Shade Tree Developer at http://codebetter.com/jeremymiller.
This entry was posted in Featured, Test Driven Development. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • tim

    Good article.
    I mho there’s a difference between ttd and using unit tests. That’s where most discussions and confusion occur.True tdd meaning wrting your test first and let the tests guide your design is somewhat difficult to master and practice. Writing good unit tests, modifying your design to easily accomodate these tests is an equal good pratice. My personal choice goes out to the last.

  • Carlton

    Actually, I think game development does have a good ROI for TDD. I am aware of one studio that INSISTS on TDD because the code quality improves their ability to deliver faster. Check out this CTO blog on agile game development. (http://www.agilegamedevelopment.com/blog.html)

  • http://blog.troytuttle.com Troy Tuttle

    Thanks for the refreshing, pragmatic description of TDD. I’ve always been suspicious when high-paid agile consultants tell you that “true TDD” will be uncomfortable and/or painful because it isn’t natural to the average developer. I think this is bunk. Other than when I had to get my mind around the concept of TDD in the beginning, TDD has always been a natural fit for me.

  • http://codebetter.com/blogs/jeremy.miller Jeremy D. Miller

    Maybe not the 3D rendering engine type work so much, but my impression is that a lot of the decision making in the game is scripting. I think TDD makes a lot of sense in that regard.

  • http://kentb.blogspot.com Kent Boogaart

    I was wondering the other day about domains in which unit testing wouldn’t yield good ROI. The only thing I could come up with is game development, but only because games have a short shelf life, so the long term benefits of unit tests don’t count.

  • Joseph D. Gutierrez

    I was just reflecting on this same thing. For the last week I have been using netbeans IDE to write some Ruby (very good IDE, has code completion/intellisense). I couldn’t event tell you if it has watches. This past weekend I decided to write my own acceptance tester (homegrown ala FIT). I ran into some issues where I started thinking if I needed to set some watches and find out what some of the intermediate states were. I stopped myself and deleted not only the source code, but also the unit tests. It took me about another hour and a half and I had a very good design with just the right amount of granualrity in my unit tests. It just flowed. I look at my previous attempt as a spike to try and capture some of the problem domain. It worked very well. IMHO if I would have kept trying to make the previous code work, not only would it have been brittle, but I think my unit tests would have been also. After getting the design ‘right’ It made me realize that I’m dealing with a FSM. That is the next level.

  • http://devlicio.us/blogs/derik_whittaker/default.aspx Derik Whittaker

    I agree with your statement ‘I have yet to work in a problem domain where I couldn’t effectively use TDD for most of the code I wrote’. I know that since i started following TDD about the only thing i could not effectively test are devices such as printers, cash drawers, etc.

    Excellent post

  • http://www.lostechies.com/blogs/joe_ocampo Joe Ocampo

    >Are you having to use the debugger quite a bit on your own code?

    This is dead on. Before TDD I relied heavily on my debugging experience. Now since I have been using TDD, I forget sometimes that there is a debug mode. I focus so intently on making the test pass, and the design of my code that when I do debug, I know something is terribly wrong. It’s more of a smell now.