In TDD Red is not ‘does not compile’

Posted by Ian Cooper on March 26, 2010

I see a bit of a growing meme that the red step in Test Driven Development‘s (TDD) red-green-refactor cycle means that your test will not compile because there is no code to implement it yet.

Bunkum!

The red step is necessary because of the possibility that your test itself has an error in it. We face something of a chicken-and egg problem with a test. We understand that we should write a test before writing code, but we cannot write a test before writing our test code with hitting a recursion issue. So we say that we write a test before writing any production code. However, our test code is a valuable as production code, so we need to prove its correctness too.

How do we do that?

By making sure that when our test fails as expected. The four phase test model (setup, exercise, verify, teardown) and the arrange-act-assert pattern all say the same thing – put the system is a known state, exercise the change we are testing, and verify the new state of the application. So by stubbing the operation under test to put the application into an invalid state after it is called, we cann prove that our test will fail. This check of the test itself makes the bar go red in our Test Runner. Hence the ‘red’ name for this step.

A corollary to this is that you should author your tests so that you can simply prove success-failure of the test. Tests that have conditional logic for example are bad tests – you should know what your test is doing. So getting a red step to fail easily is generally also a good marker that we are indeed testing a unit. If getting a red is really hard, there is probably an architectural smell.

There is a temptation to skip the red phase and go straight to getting to green. The risk here is that you have a false positive – your test would always pass. Kent Beck talks about the model of driving in a different gear when doing TDD. Skipping the red phase can work if you feel able to drive in a high gear – the road ahead is straight, wide open and flat and you want to eat miles as fast as you can. However as soon as you hit a steep downhill gradient, bends or traffic, you need to shift down again. Usually the indicator here would be that you find your tests would probably always have succeeded when you get to green. However, having regularly encountered mistakes in my tests using the red step I would avoid driving in high gear without thinking about it.

But remember red is not ‘fails to compile’ it is ‘test fails as expected’.

 

About Ian Cooper

Ian Cooper has over 18 years of experience delivering Microsoft platform solutions in government, healthcare, and finance. During that time he has worked for the DTi, Reuters, Sungard, Misys and Beazley delivering everything from bespoke enterpise solutions to 'shrink-wrapped' products to thousands of customers. Ian is a passionate exponent of the benefits of OO and Agile. He is test-infected and contagious. When he is not writing C# code he is also the and founder of the London .NET user group. http://www.dnug.org.uk
This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://jimmybogard.lostechies.com Jimmy Bogard

    Another corollary to that one – red also means assertion failure, not exception thrown. I hate the “throw new NotImplementedException” as a test failure.

  • http://codebetter.com/members/Ian-Cooper/default.aspx Ian Cooper

    @Jimmy Too true. You need to know your assertion will fail before you make it pass. Not just skip the assertion altogether

  • http://derickbailey.lostechies.com Derick Bailey

    nice – had a similar post just yesterday… red/gree/refactor for the right reasons.

    totally agree with what you’re saying, Ian.

  • http://www.jacopretorius.net Jaco Pretorius

    Interesting – I don’t really have a problem with ‘throw new NotImplementedException’ as a test failure. At this stage I’m concerned about the test, not the implementation. As long as it compiles and I get a failure on my test I’m happy. Next step is to get the test to go green in the easiest way – usually one or 2 lines.

    Then I move on and write another test. I don’t really see the problem with the ‘throw new NotImplementedException’ bit…

  • http://codebetter.com/members/Ian-Cooper/default.aspx Ian Cooper

    @Jaco The problem is that you only know that your test fails if your code throws an exception, not if the state after the operation is invalid. You may believe that, but you have never tested it.

  • fschwiet

    Theres a similar gotcha. Sometimes I’ll write code to make the test pass, and it doesn’t pass. I look into it some more, and its a test bug not a implementation bug. I fix the test bug, and the test passes.

    This is dangerous. When this happens, need to be careful to back out the code change to verify the test still fails. Otherwise you may still have another test bug (false positive) that may be hiding a code bug as well.

  • Greg Symons

    When writing a completely new feature, something that is orthogonal to everything else in the system, I’ll often first start with a NotImplemented exception (or similar, depending on the language I’m working in), just so that I can see that I can compile the code and run through all the set up. This is especially important in dynamic languages… I’ve often written code in these languages that looks right and will “compile”, only to have some silly runtime error take it down before it even executes the test. Once I make sure I can get to the “act” part of the test, I then put in an implementation that I know will fail to check the “assert” part of the test. Only after I’ve seen the entire test run in red, do I actually fill in an implementation that could work.

    That way, when the test goes green, I can feel confident that both the test and the implementation work the way I expect them to.

  • http://fernandozamorajimenez.blogspot.com/ Fernando Zamora

    Had a test that went green from the go. I needed a red (failed) test though. Once I looked further into it reveiled a test that I did not write. Once I wrote the other test and put it through the process RGR, I went back to the original test and made it red. Then I wrote the code to make it green. I used to be a skeptic about Red Green Refactor a few years ago but experience with false positives has made me a believer.

  • http://www.jacopretorius.net Jaco Pretorius

    @Ian I still don’t see your point – for example, let’s say I’m using TDD to test some calcuation and for the specified inputs I’m expecting an answer of 5.

    So I will have Assert.AreEqual(answer, 5);

    My test will fail if I throw a NotImplementedException. My test will also fail if I return anything other than 5 – I don’t care about the implementation detail – if anything other than 5 gets returned the test will fail.

    Plus this is only an intermediate step right? 2 Seconds later I will go to the implementation and change it to say ‘return 5;’

  • http://melioratingmonkey.blogspot.com/ MM

    @Jaco – There is a difference between failing because of the wrong return value and failing because the SUT throws an exception. Let’s say that in your test code you forget the assert but in your SUT you throw NotImplementedException. So now you have the Red part satisfied even though you have a bogus test. Furthermore, all you have to do to make the test green is delete/comment out the line that throws the exception. Obviously you would never do this on purpose but I could eventually see it happening and it could lead you to think that you’ve implemented a feature that you haven’t.

  • http://blog.kjempekjekt.com Torbjørn Marø

    I usually go through several red-green-refactor cycles for each test. First I may instantiate an object of a class that does not exist yet. I’m satisfied with the compile error as a red test, and I don’t allow myself to add any more code to the test at that point. Instead I add enough application code to make the test green – adding the empty class.

    Next I might have to call a method. The method don’t exist, causing a compile error. So I stop and add the method. Green again.

    And now it may be time to add an assert. It may or may not give me a compile error. If it does I implement enough to get it to compile, but I also need to see a red test that does compile at this point – so I agree with you there.

    Just wanted to share, and remind you that a test can be grown in several steps, and that the compiler can have it’s use in the cycle.

  • http://MaggiePlusPlus.com Maggie Longshore

    A failing test is very important. I had a passing test that I knew should have failed. Looking further I found a test helper method that had been modified (by mistake I believe) so that the tests using it could never fail.

    I fixed it and uncovered several failing tests. If the devs were all using red-green-refactor it would have been uncovered months ago.

  • http://jamespeckham.com james peckham

    @Jaco

    The idea here that everyone is trying to tell you is something like this…

    let’s say you have

    int value = 0;
    public void foo()
    {
    throw new NotImplementedException();
    }

    then you change it to
    int value = 0;
    public void foo()
    {
    value = 0;
    }

    your test now passes, but it would still pass if you had this code:
    int value = 0;
    public void foo()
    {

    }