Testing Granularity, Feedback Cycles, and Holistic Development

So far this year I’ve done a greenfield development project, made changes to a legacy application, and am about to take over development on a third product.  The new code was built with Test Driven Development and Continuous Integration from the beginning.  It’s relatively simple to create granular unit tests and work in a TDD manner.  Our feedback cycles between coding and testing are fairly short.  I can work confidently with the code because I know I can verify the correctness of the code at any time.  Compliance with the requirements via automated acceptance tests is a different matter, but we’ll get there. 

By contrast, the older code wasn’t built with TDD or CI and it shows.  We’re in an unfortunate situation with the legacy code.  I’ve been employing the dependency breaking techniques from Michael Feather’s book on legacy code to write new code with unit tests, but the interaction with existing code has been the source of most of the errors.  In order to do any kind of integrated testing I effectively have to migrate code changes over to a Virtual PC image via an MSI package (don’t ask, I don’t want to talk about it) and take several manual steps to cleanse the test environment.  That sluggishness in the feedback cycle was extremely frustrating and helped stretch a miserable project by a couple of weeks.  The work went a lot faster when I partially automated the code migration with some shortcuts.

One of the sharpest contrasts between the greenfield code and the legacy code has been the length of time between doing something and the subsequent verification of that something.  Rapid and continuous feedback is one of the best and most important attributes of Agile development.  Without constant and efficient feedback mechanisms, the coding work has many more chances to go off in the wrong directions.  Moreover, the adaptive manner of doing work in Agile projects is dangerous without the constant corrective steps generated from feedback.

Test Small Before Testing Big

Things always go better when you code and test in small chunks.  The difficulty in diagnosing a test failure is geometrically proportional to the granularity of the test.  Granular unit tests are easy to debug.  Coarse-grained integration tests involving a dozen classes and external resources are many times more difficult to handle when they fail.  Do the granular unit tests first to mitigate the “debugger hell” of the large integration tests.  Don’t even attempt to execute a large integration test until you are relatively certain that each individual piece of code works in its own set of unit tests.  The product that I’m working on for the rest of the year has very good coverage from integration tests, but rather anemic coverage from unit tests.  We’ll be able to make changes in the code with confidence because of the integration test coverage, but fixing any regression bugs is going to be miserable because of the spotty unit tests.  I’m not writing this post to criticize my peers, I just think that writing the unit tests first would have made their development go faster.

One of the things Feathers mentions in his book is that tests are also a way to preserve behavior in the face of later changes.  A rigorous unit test suite should tell you exactly when and where your new code changes break existing code.  Coarse-grained tests may only tell you that you’ve broken something — somewhere.

Unit Tests Should Be Fast

I want to write code in a rapid cycle of “write a unit test, make unit test pass, refactor, repeat.”  We use TestDriven.Net to quickly compile and run individual unit tests from the IDE as we’re constructing the code.  In a normal coding day I may literally be running several hundred unit test runs, each with an accompanying compile of the code.  A slow compile step or post-compile step (watch what you’re doing in post-build steps!) can bring coding velocity down to a crawl.  The product I just inherited has up to a 4 1/2 minute compile time for the whole solution and a minute plus for running an individual unit test.  That compile time is nothing but overhead and friction.  We’ve identified a strategy for consolidating the code to bring the compile time down, but it won’t be fun.  The code consolidation may be miserable and time consuming, but say I really do run unit tests at least a hundred times a day.  If I can cut just a half minute off the compile time I’ve saved almost an hour of thumb twiddling a day.  Over time the overhead charge of the code consolidation will rapidly pay for itself many times over in improved developer efficiency.

Optimizing Build Time

If you’re doing Continuous Integration, you want the build to be as fast as possible for quick feedback.  Long build times are an insidious form of project friction.  The whole point of doing CI is to know the current code revision is in a valid state.  SDTimes is running an article on the challenges and tools for streamlining the build process for larger projects.  There is a great quote in the article from Pragmatic Programmer Andy Hunt – “Teams that want to be more agile are headed for a train wreck if they have long build times; they’ll need to find ways to build all or part of the software more frequently to get the kind of continuous feedback that helps agile teams move quickly.”

My colleague and I have had the opportunity to present on Continuous Integration a couple of times this year.  We spend a little bit of time talking about strategies to optimize the build process.  Typically you would expect the automated testing to be the bottleneck.  Integration and blackbox tests can be relegated to a secondary staged build, but unit testing is pretty well mandatory for adequate feedback.  Testing database access, user interface screens or web pages, and SOAP calls are obvious culprits for slow tests.  I know that web services are the greatest invention since fire, but they’re absolutely unusable for quick unit testing.  Here is some of my recommendations for dealing with slow unit testing.

  1. Aggressively mock the database during unit tests (don’t mock ADO.NET directly though, that’ll hurt).  A domain model approach with “Persistence Ignorant” classes is even better for optimizing unit test execution times.  The major point being to leave yourself a way to test business and control logic without the database.
  2. Use some sort of Model View Presenter architecture for user interface code, a.k.a. “The Humble Dialog Box.”  Isolate as much user interface code as possible from whatever GUI framework you’re developing with.  Slice the actual view code as thin as possible and mock the view when you test the controllers.
  3. Always use the Dependency Inversion Principle when accessing web services in code.  Leave a mechanism to mock out the web services while testing client code.
  4. Decouple the actual functionality of a web service from the SOAP message transport.  The web service should be a thin wrapper delegating to a plain old object.  If you can bypass the SOAP serialization and the HTTP calls your unit tests will be much faster.  I don’t know about the J2EE world, but in .Net development the IIS web server is a huge pain in the ass during unit testing.

The point I’m trying to make with this list is that the primary way to optimize the build process is by making appropriate application design and architecture choices.  An experienced TDD team will purposely design an application with ease of testing and build automation as a first class consideration.  Just as an aside I have my own private “Chicken or Egg” debate.  I don’t know if your code is testable because you’re doing TDD or if you write testable code in order to do TDD.  Either way, TDD and CI will be miserable experiences unless a team builds up a feel for creating testable code.

Holistic Approach

One of the most profound philosophic shifts in Agile development may be a holistic view of software development.  The real goal of my work is to get the code into production, not just over to the testers on time.  Blurring the responsibilities and roles on a software development team leads to better informed individuals that can look at the bigger picture.  Getting code into production requires testing to remove defects, so I start building code to be easier to test.  The team’s velocity is affected greatly by the build process, so I architect the application to be both easier to deploy (X-Copy!) and faster to build with automated testing.  Designing for testability will very often require more coding than you would do otherwise.  The extra code in unit test suites is an obvious source of objections to doing TDD.  In the end, the extra time spent writing unit test code seems to consistently pay off by getting the code to production quality faster. 

One of my colleagues made a simple, profound statement to me last week that “TDD drives you to a different approach.”  Keeping in mind that TDD is only the means and not an end in itself, my reply is that any technique that gets me to done faster is worth writing more or just different code.

I think that unnecessary specialization of personnel in software development causes a tremendous amount of damage in our industry.  How in the world can a Non-Coding Architect truly account upfront for ease of testing, building, and deployment if they’re never involved in the downstream project activities?  I honestly think that being much more involved in the entire software process has made me a better software designer.

The “Idea Wall”

A nice practice I’ve used on a couple of Agile projects now is the “Idea Wall.”  Pick some visible spot on the wall in the team area, and post any sort of idea for improving the project infrastructure on “Post-It” notes or index cards.  Distributed teams can use a Wiki instead.  Capture any issue that is slowing down the team like slow compiles, missing environment setup, or nice to have items like code statistics or better testing tools.  When any developer is idle they can pick something off of the idea wall.  If your team is doing pair programming, you can use the idea wall as a way of collecting non-coding tasks for the “exposed” person on odd-numbered teams.  If the idea wall starts to get too cluttered, it may be a good sign that you’re incurring unnecessary friction and you really need to invest in the build infrastructure.  Don’t tolerate inefficiency in your build or development.  Sometimes you really need to stop and work on the build before you write any new code.  Project managers need to understand the necessity of these kinds of chores.


If you actually finished reading this post, thank you for putting up with a bag of incoherent rambling.


About Jeremy Miller

Jeremy is the Chief Software Architect at Dovetail Software, the coolest ISV in Austin. Jeremy began his IT career writing "Shadow IT" applications to automate his engineering documentation, then wandered into software development because it looked like more fun. Jeremy is the author of the open source StructureMap tool for Dependency Injection with .Net, StoryTeller for supercharged acceptance testing in .Net, and one of the principal developers behind FubuMVC. Jeremy's thoughts on all things software can be found at The Shade Tree Developer at http://codebetter.com/jeremymiller.
This entry was posted in Continuous Integration, Test Driven Development. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://www.jeffreypalermo.com Jeffrey Palermo

    I agree about non-coding architects. Somehow the UML diagrams just don’t match up with what the software should really look like. Also, How many UML diagrams that you’ve seen call for dependency injection for external dependencies (like the database)? I’ve seen domain libraries _designed_ to reference a particular data access layer. That’s a good way to kill logic portability.

  • http://codebetter.com/blogs/jeremy.miller jmiller

    Darrell – You know you’re my best editor;-)

    Josh, thanks for the tip.

  • http://flimflan.com/blog Joshua Flanagan

    “migrate code changes over to a Virtual PC image via an MSI package (don’t ask, I don’t want to talk about it) and take several manual steps to cleanse the test environment”

    Maybe this is what you were referring to, but maybe you aren’t aware of it: Microsoft
    Virtual Server exposes a complete COM object model for doing this exact sort of thing (automatically wiping environments, etc). Virtual Server images are compatible with Virtual PC images too.

  • http://codebetter.com/blogs/darrell.norton/ Darrell

    It was Mike Clark that said the Train Wreck comment, not Andy Hunt. Mike Clark’s weblog is at: http://www.clarkware.com/cgi/blosxom

  • http://codebetter.com/blogs/darrell.norton/ Darrell

    Good post!

    Test small before big works for projects that are built for testability, but sometimes all you have is a group of known “correct” results that are integration tests. I’ve used those to refactor code to make it more testable in places while ensuring the app still works according to the known integration tests. Yep, it’s a pain, but better that than stopping the users.

    For build times, if you setup objects correctly and factor out database calls, network calls, etc., then your tests will almost always run fast. If they don’t, your code is not performant and it’s not really the tests’ fault. :) And by setting up objects correctly that includes lazy instantiation where necessary so that tests on 1 method that don’t require an expensive creation process don’t call it, for example.

    For Non-Coding Architect, it depends on the type of architect. I don’t think Rich Turner does much coding, but I damn sure want to work with him as an “Architect”! :)

  • http://exceptionz.blogspot.com/ Maruis

    Your post has a lot of good advice. It is true, it doesn’t matter how much pressure and stress there is on a project, if the CI or some development or release process needs improvement. Then it would be best to first solve these problems. Any case, thanx for the thoughts. Made for good reading…