A Taxonomy of Tests

This was originally a series of posts at my old blog a few days after I recovered from the midnight showing of “Revenge of the Sith.”

I made the mistake of posting on writing good unit tests while suffering from “Jedi Flu” and Jeffrey Palermo took me to task for not defining a unit test first, so here’s my attempt at describing the various types and approaches of tests I’ve seen or used on agile projects. Keep in mind that I’m a developer first and only an occasional architect, “BuildMaster”, or tester second. The first thing to keep in mind is that agile processes don’t change the fundamentals of software development. It is still best practice to test the code at multiple levels of integration (the old V-curve idea). It’s great to mock the database in your unit tests, but the database is still there and just itching to get out of synch with your middle tier code.

A Taxonomy of Tests

Roughly, I would divide tests into:

  1. Unit tests
  2. Acceptance tests
  3. Intermediate/System/Integration tests, written by developers
  4. Other: smoke tests, regression tests, performance tests, miscellaneous validation

Testing Strategy

The answers will vary from project to project due to manpower constraints and the nature of the development, but any and every project is going to have to answer these questions below.

  • Who writes what tests?
  • Who automates the tests, and how?
  • Where and when do the tests run?
  • How do we want to organize the test suites?
  • How much testing is enough?

Sleeping with the Enemy

One thing I’ve learned that does not vary; it takes a great deal of cooperation and mutual respect between the testers and coders. Working in the same room is ideal. On projects where the tester and developers are separated by either organization or location, the development effort will suffer. For that matter, both developers and testers need quick and easy access to whoever holds the reins on requirements. At a bare minimum, don’t look at testers as the enemy.


Unit Tests

One of the truly great advantages agile development has over traditional waterfall projects is in team dynamics. There is far more interaction and communication between disciplines. Just being on the project at the same time is an advantage over “throw it over the wall” processes. Everybody’s common goal on an agile team is producing working software. Not code, tests, UML diagrams, or Gantt charts, but working code that meets the customer’s expectations.

At the most granular level is the unit test. Unit tests are written by developers to verify, or in Test Driven Development specify, the functionality of a very small piece of code to the satisfaction of the developer. Usually a unit test works on one class or a small cluster of classes at a time. One of the key attributes of a unit test is the ability to test a small unit in isolation. We want to prove that the unit of code itself is working without corruption or interference from other classes.

Purposely designing code in order to create isolated unit tests is part of a successful Test Driven Development effort. Mocks and stubs are an invaluable tool to create the isolation. Some people insist that all external dependencies (databases, Active Directory, web services, etc.) be mocked out in unit tests. My best advice is to consciously quarantine inconvenient things behind gateways with an abstracted interface. Use Inversion of Control to minimize dependencies from the majority of your .NET code to these gateways. When you can’t eliminate a dependency, use Dependency Injection to provide a mechanism to replace the real thing with a mock or a stub. I have had very positive (well, mostly) experience with the Model-View-Presenter (MVP) pattern described by Martin Fowler as a strategy to maximize the testability of user interface code.

Unfortunately, some units of code only exist to manipulate the database or interact with infrastructure services like messaging. Not to mention that you still have to unit test the UI screens. MVP is not enough in and of itself to guarantee a properly functioning user interface (trust me on this one). You can either marginalize this code by not writing tests and hope for the best (or let the acceptance tests take care of it), or bite the bullet and write a battery of tests that are connected. My advice is to put some ketchup on the bullet and just do it.

I do prefer to separate connected and non-connected unit tests into separate test fixture assemblies. I generally lean on only the non-connected tests during refactoring or new development. I do think it is absolutely necessary to run the integrated unit tests in both your check-in procedure and the automated build.

There is a school of thought that says if you really unit test well enough against the mocks, integrated testing is unnecessary. The NEO project is the best attempt I’ve seen anybody use to actually approach this ideal. I still think this philosophy is unattainable and possibly dangerous.

I’ve been in some heated arguments on the merits of mocking ADO.NET when testing custom data access classes. I mostly vote no to mocking ADO.NET. In this case, the unit of valuable work is manipulating the data in the database. Who cares if the code sets the “expected” parameter values when the real stored procedure has been changed in the database behind your back? In this case, the effort to value ratio is all wrong. Of course, when you’re a consultant onsite at the client you work according to their goofy development standards and politely say “Thank you sir, may I have another?”

Acceptance Tests


From Ward Cunningham’s wiki: “A formal test conducted to determine whether or not a system satisfies its acceptance criteria and to enable the customer to determine whether or not to accept the system.”

The acceptance test is exactly what it sounds like, it verifies that the implementation of a user story works as the customer requires. In a literal interpretation of XP theory, the customer writes the acceptance test. In reality, the acceptance tests are going to be written by some combination of the business partner, testers, business analysts, or product managers. It really doesn’t matter to me, as long as they are done in a timely manner with some degree of correctness. You’ll never have the ideal XP customer, but it doesn’t necessarily mean the project is hopeless.

It is a huge advantage when acceptance tests are approved and ready during coding. User stories, use cases or huge paper System Requirements Specification documents — the best and most effective requirement is a cut and dried acceptance test. When the coders can (and do) easily execute the acceptance tests themselves, the bug counts will go down. The development team as a whole can use the acceptance tests as the “Red Bar” metric to watch during an iteration. A user story is “done” only when all the finalized acceptance tests pass. James Shore has a good post on acceptance tests as requirements here.

Developer/Tester Cooperation

Testing features during an iteration as soon as the code dries is a tremendous advantage. It’s much, much simpler to fix bugs that were just coded before the memory of the code fades and the bug is covered up my much more code.

My current project has not had acceptance tests written during iterations, and it has clearly hurt us in terms of wasted effort and misunderstood requirements. Never again.

Organizational issues between testing and development are not specific to agile development, but the desire for rapid iterations will expose any friction between requirements, code, and testing. Moving fast without sacrificing testing requires the ability to cycle code quickly between development and testing environments. Code that is easy to test helps quite a bit, too. Developers have to support the testers with the mechanics of testing. Both testing and development can benefit from a good Continuous Integration (CI) strategy. Testers should be involved in the CI effort. I feel very strongly that it is the developers’ responsibility to ensure the test environment is in a valid state. “BuildMaster” folks are a desirable luxury, but I always want visibility into the test server environment.

Automating Acceptance Tests

To paraphrase Ron Jeffries, “Unit tests are for coding right, acceptance tests are for writing the right code.”  Acceptance tests are a good and necessary thing because they are effectively the requirements in an agile process.  Ideally, the acceptance test should be an unambiguous statement to the proper functioning of the software.  If the acceptance test passes, the requirement is satisfied.  As far as testing is concerned, an automated test that runs often and visibly is far more valuable than a slow manual test buried in a test plan. A common objection to iterative development is the need for so much regression testing. The best answer to this perfectly valid objection is to automate as much as you can.  Enterprise applications are rarely finished; they’re abandoned when they are judged to be too expensive to change. Automating acceptance tests is a great way to extend an application’s lifecycle by reducing the cost of regression testing. Also, to paraphrase a former colleague — “If your tests aren’t automated, how do you know they’re passing?” Think on that one.

Who writes the test is one decision. Who automates the tests, and how, is a different decision. Hopefully you’re lucky enough to have testers with programming or scripting abilities. Even if you do have technical testers, plan on the developers assisting the testers with test automation. I’m not a big fan of the mechanics of FIT and its derivatives, but it is definitely a step in the right direction for acceptance testing (I love the concept, but I hate the implementation of NFit). The FIT model allows for writing acceptance tests in near English, with some non-trivial work behind the scenes to complete the wiring for the custom FIT fixtures.

Conventional wisdom says that acceptance tests are black box tests that test the system end-to-end. This doesn’t necessarily have to be the case. I’ve seen both Brian Marick and Brett Pettichord write about using white box tests for acceptance tests, especially for user interfaces. Using Service Orientation principles to create a cohesive service layer can be a great aid for automated acceptance testing. I think (N)FIT chokes on user interfaces, but it is perfect for driving an isolated service point (note I didn’t necessarily say web service) in an automated test. These white box tests can be much easier to setup and execute. Some flavors of MVC architecture allow for slipping in a screen proxy in place of a real UI view (I’ve had mixed results with this). Again, it is not a “perfect” black box test, but it goes a long way towards validating the behavior of the system.

Good testers can work with developers to write these automated whitebox tests. It does take pretty good communication and cooperation between coders and testers to take advantage of these opportunities for test automation. It is definitely advantageous for the testers to be familiar with some of the inner workings of the system. I definitely think agile development starts to blur the line between developers and testers, and that’s not an entirely bad thing.


Everything Else


While the majority of your focus on an agile project is on either unit tests or acceptance tests, there are quite a few other types of useful tests that fall through the cracks. All of these types of tests can and should be part of your continuous integration strategy.

Developer Written Partial Integration Tests

Bigger than a unit test, but smaller in scope than an acceptance test. That’s a big mouthful for a category, but I’ve never seen any two development teams that use the same name for this broad category of tests. For multiple reasons, most of my team’s unit tests are disconnected — i.e. databases, web services, and the like are mocked or stubbed. I want to test each class in isolation, but they have to interact with the real services too. A secondary battery of developer tests that test partial code stacks are very advantageous in uncovering problems in the actual interaction between classes or subsystems. I’ve known some XP developers who actually emphasize these semi-integrated tests more than the disconnected unit tests. I like to keep the types of tests in separate test assemblies, but that is just preference. These are really hard to describe, so here’s a couple of ideas I’ve used or heard of —

  • Testing a view and controller or presenter together. I would still try to mock out the backend business services from the controller in order to focus on testing the integration of view and controller. This is especially important in the new Model-View-Presenter mode of UI development.
  • Testing from a UI controller down to the persistence layer.
  • Test from the service layer down. SOA is surely overhyped and overused, but it presents some great opportunities for automated testing. Because the .Net SOAP serialization is a bit brittle I always put automated tests in for the web service proxy class communicating with a local web service. It’s also good to test the service code without IIS being involved.

Environment Tests

Environment tests verify that a build or test environment is fully connected and correctly configured. I helped do a presentation on continuous integration to the Austin Agile group last week and we spent quite a bit of time talking about this very topic. The gist of the slide was preventing lost time by testing teams by uncovering disconnects or failures in the testing environment at code migration time. Nothing is more aggravating than reported defects that end up being merely a mistake in code migration. I spent a week one time getting grilled because a data interface wasn’t working correctly. It turned out that the middleware team had shut down the webMethods server in the test environment and not told the development team. Putting aside the total incompetence of everybody involved (including me), had an automated environment test been in place to verify the connectivity to webMethods, a full week of testing would not have been wasted.

I had some periodic trouble on a project last year with maintaining various copies of the system configuration (database connections, file paths, web service locations, etc.) for development, build and test environments. To combat this problem in the future I’ve actually added some support into my StructureMap tool for doing environment checks and configuration validation inside of an automated deployment. We sometimes move code a lot faster in agile development and it provides a lot of opportunity to screw things up. These little environment tests go a long way towards keeping project friction down.

I got the name for this off Nat Pryce’s post here.

Smoke Tests

Smoke tests are simple crude tests to exercise the system just to see if the code blows up. They don’t provide that much benefit, but they’re generally low effort as well. On my current project we’re replacing a large VB6 component with a new C# equivalent. One of the things we did to create a safety “tripwire” effect was to run a huge amount of historical input XML messages and configurations through the new C# engine just to find exceptions. We found and fixed several problems from edge cases and some flatout bugs. It was definitely worth the effort.

Regression Tests

My last two projects have been more or less rewrites of existing legacy code. Oftentimes hairy legacy code isn’t understood very well by the organization (hence the rewrite). Twice now I’ve had to jury-rig the legacy code to create repeatable automated tests that simply prove the new code creates the same output for known input. It’s not entirely a good test because it has no connection to the intent of the code, but both times it found problems. The irritation level was high because the legacy code wasn’t pretty or easy to get running, but otherwise unknown defects removed and fixed made it worth the effort. Automate the regression tests in the build if you can. If nothing else, there are always business processes and requirements buried in the code that aren’t known or documented anywhere else.

“Build Time” Validations

There are always some aspects of the system that the compiler and maybe even the unit test library can’t validate. Use the automated build as an opportunity to create “build time” checks for potential problems or enforce coding standards. Here’s some ideas of things to enforce in the build:

  • A compiler warning or “TODO” threshold. I’ve heard of people making a build fail for this.
  • A limit on ignored unit tests
  • SQL validator. Gotta keep the sql external to the code somehow, though.
  • O/R Mapping. We’re starting to adopt NHibernate for persistence. One of the first steps will be validating the mapping in the build
  • Automated code coverage is kind of a test of your tests
  • Static code analysis

There’s probably more, but that’s all I can think of. I’d love to hear other ideas.

About Jeremy Miller

Jeremy is the Chief Software Architect at Dovetail Software, the coolest ISV in Austin. Jeremy began his IT career writing "Shadow IT" applications to automate his engineering documentation, then wandered into software development because it looked like more fun. Jeremy is the author of the open source StructureMap tool for Dependency Injection with .Net, StoryTeller for supercharged acceptance testing in .Net, and one of the principal developers behind FubuMVC. Jeremy's thoughts on all things software can be found at The Shade Tree Developer at http://codebetter.com/jeremymiller.
This entry was posted in Test Driven Development. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://www.backlinkschecker.ws/backlinks/buylinks.html Purchase High PR Backlink

    I am really enjoying this site

  • http://davidhayden.com/blog/dave/ dhayden

    I second that. Excellent post, Jeremy.

    Well written and a lot of great information. Welcome to CodeBetter!

  • http://codebetter.com/blogs/brendan.tompkins Brendan Tompkins