The 8th Phase

I once posted a semi-serious post entitled The 7 Phases of Unit Testing. The phases are:

  1. Refuse to unit test because “you don’t have enough time”
  2. Start unit testing and immediately start blogging about unit testing and TDD and how great they are and how everyone should do it
  3. Unit test everything – make private methods internal and abuse the InternalsVisibleTo attribute. Test getters and setters or else you won’t get 100% code coverage
  4. Get fed with how brittle your unit tests are and start writing integration tests without realizing it.
  5. Discover a mocking framework and make heavy use of strict semantics
  6. Mock absolutely everything that can possibly be mocked
  7. Start writing effective unit tests

I think the cycle I went through is extremely healthy – unit testing is something best learnt from practice and is something you refine over time. Judging by the comments on the original post, a lot of you agree.

Recently though, I’ve felt like adding another stage:

8 . Sometimes the best tests aren’t unit tests

After awhile, it becomes obvious that some tests are significantly more meaningful when you expand your scope – say to include hitting an actual database. Narrow unit tests and wider integration tests can always work together; but I’ve found that, in some cases, more comprehensive tests can replace corresponding unit tests. This may not be a proper, but it is practical.

When I say unit test, I mean the smallest possible unit of code – generally a behavior. Most methods are made up of 1 or more behavior. I’d say that you shouldn’t have too many methods with more than 6 behaviors (as a rough goal). As an obvious example, in NoRM this method helps identify the type of the items in a collection:

public static Type GetListItemType(Type enumerableType)
    if (enumerableType.IsArray)
        return enumerableType.GetElementType();
    if (enumerableType.IsGenericType)
        return enumerableType.GetGenericArguments()[0];
    return typeof(object);

Clearly, this method is a good candidate for 3 or 4 unit tests (one when the type is an array, one when it’s a generic, one when its something else, and maybe one when its null).

As your code moves closer to the boundaries of 3rd party components, the value of unit testing may suffer. You’ll still get the benefits of flushing out coupling and enabling safe refactoring (which shouldn’t be underestimated), but you’ll likely miss out on making sure things will work like they should in production. The solution can be to expand the scope of your tests to include the 3rd party component.

The most common example is database code. Testing a Save method by mocking the underlying layer might work, but there’s value in making sure that the object actually does get saved. That isn’t to say that a single test that hits the database is good enough – your Save method might be made up of multiple behaviors, some which are better validated with one form of testing than another.

Really, that’s one of the key things to remember as you walk down this path – don’t think that just because your method is actually saving an object that your job is done. There are likely other behaviors that aren’t being tested at all. It’s easy to abuse these types of tests and get a false sense of security. The other key is to make sure that it runs like a unit test – namely, that its fast, doesn’t require any manual setup, and isn’t dependent or doesn’t break any other test.

Lately, I’ve seen interest in using in-memory databases for this type of thing. The benefit is that they are super fast and don’t leave stale data. They also don’t require special setup. On the downside you still aren’t truly testing the most fundamental behavior of your method – that an object will be saved to the database in production. Even with the best O/R tool I’ve seen code work against one database but not work against another – due to a bug on my part. Writing a script that can automatically and quickly setup and teardown against the final database, and having your team members set up a local database, may or may not work for you (it’ll depend on the nature of your team and your system).

Ultimately, the most important thing is that you have automated tests which aren’t a nightmare to setup, maintain or run. Integration tests have more dependency and thus are more fragile, but can be an efficient way to verify correctness.

This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.

9 Responses to The 8th Phase

  1. Ben says:

    I’ve largely settled on unit testing for behaviors and bdd-style tests for end to end testing (using mspec and a separate integration “db”).

  2. Martijn Verburg says:

    Great Article! I’m still no guru but as I worked through the first 7 steps I definitely started to appreciate the need for all sorts of levels of testing.

    Unit Testing made my code better on a smaller scale, but the integration testing saves me from massive real world headaches.

    Mind you, writing Integration Tests is hard, especially when talking to 3rd party resources, I’m still trying to find nice ready made stubs for things like FTP servers and financial gateways

  3. Ah, but there are at least 5 more phases after #8. I’ll give you #9 – “Blog about how your first years of testing were sub-optimal.”
    I won’t reveal the other future steps so I don’t spoil your personal experience. #sarcasm

  4. “Integration tests can be run nightly by the CI server. To make this feasible, though, the two types of tests need to be in separate assemblies (in .NET).”

    We’ve got them in the same assembly*, but are using seperate TestFixtures and using Categories to allow NUnit to run one set or the other,

    [TestFixture, Category(Test.UnitTest)]
    public class MyClassUnitTests…

    [TestFixture, Category(Test.IntegrationTest)]
    public class MyClassIntegrationTests…

    then invoke unit-console.exe with either the /include=”UnitTest” or /include=”IntegrationTest” option.

    *We were going to split them up but then we would have 2 test assemeblies per production assembly and that made the Solution more ‘noisy’.

  5. I agree with MM on this – we need both types of tests (unit and integration).

    To me the most important thing is to separate integration tests (relatively slow) from unit tests (relatively fast). Unit tests can be run after every refactoring, by a Continuous Integration server on every revision control check in, etc. Integration tests can be run nightly by the CI server. To make this feasible, though, the two types of tests need to be in separate assemblies (in .NET).

  6. Rob says:

    Ah enjoyed that list before. Been through a few of those steps myself.

    We’re currently on Phase (x). BDD. Having said that our tests are now much more readable, and understandable. We thankfully mock much less (if you’re mocking a lot of things maybe you’re trying to test too much, or there’s too much responsibility for that class).

    We’re still learning how to handle data setup though. Especially given how complex our Model is. Inmemory db’s are definitely a good way forward though.

    InternalsVisibleTo we use a fair amount, but only because our test code sits in another project, not as a change from something being private.

    I’d still like to somehow store test code alongside implementation code, for ease of discovery, and filter it out at compile time (somehow!).

  7. MM says:

    i get what you’re saying but this still strikes me as too much of an “either/or” debate when i don’t think it should be. why shouldn’t we be striving for both types of tests rather than trying to replace one with the other? we should have lots of automated unit tests and lots of automated integration tests. one of the biggest problems i’ve been facing with trying to convince people to do “proper” unit testing is that they think it somehow devalues the importance of of integration testing. they’re both really, really important so why don’t we just say that we need both?

  8. Corey Haines says:

    This is why I’ve moved to using the term ‘isolation test,’ rather than ‘unit test.’
    The term ‘unit’ has overloaded a bit too much, so people are nervous when you expand the scope, and the discussion derails into a focus on terminology. The scope of what I isolate in my tests can vary according to what makes sense based on my experience testing that type of area.

  9. Billy Stack says:

    Great post, we have pretty much went through the 8 phases you mention!

    This is where we are currently at:

    We use test categories and physical assembly separation between fast unit tests and slow brittle tests e.g. db related.

    That way we get fast unit test execution feedback – we execute all fast tests by default with the test runner.

    However other tests do have their place. Integration related tests e.g. database, service tests, IOC loading tests, ORM session factory building tests, temporal (threading) tests etc are maintained separately and are always included in our automated build process and can optionally be run instantly by the developer if needs be!

    Anyway, just said I would give my two-pence worth!