High Test Coverage Ratio is a good thing, Anyway!

Posted by Patrick Smacchia on June 7, 2009

Normal
0

21

false
false
false

FR
X-NONE
X-NONE

I heard several times that
high test coverage ratio, and ultimately 100%
test coverage, is an illusion of quality. The underlying reason is that when a
portion of code is executed by tests, it doesn’t mean that the validity of the results produced by this
portion of code is verified by tests. In other words, some code can be
covered by tests and still contains bugs.

I totally agree with this
assertion. However, I don’t agree when one claims that high test coverage ratio
is an illusion of quality. I can see several major benefits resulting from high
test coverage ratio.

 

Contract + High Test Coverage Ratio =
Correctness checked anyway

If you apply correctly Design by Contract (DbC) high test coverage pays off. DbC
means that your code contains side-effect free code whose only purpose is to
check the states correctness at runtime as often as possible. If the state correctness
is violated, a contract fails abruptly. To implement the DbC principle, we (the team working on NDepend) used
so far my good old friend System.Diagnostics.Debug.Assert(…) and we are in the process of
switching to the System.Contract API.

The trick is that if
contract validity verifications are performed during tests execution, then
tests are failing when a contract is broken. And the higher the test coverage
ratio, the more contracts are verified at test-time. By extension, one can
actually consider test code dedicated to verification (I mean lines in your
test like Assert.IsTrue(…)) as
contracts residing outside the code itself. Also, on NDepend development, we
observe that regression bugs are discovered much more often thanks to broken contracts,
than thanks to broken test. It is because contracts actually live in the code, they are much more
close to what code is doing than test themselves. Also contracts are more often
verified than tests (imagine a contract in a loop vs. a test verification
outside the loop).

One technical detail is
that TestDriven.NET discards Debug.Assert(…) failure windows by default and I explain here how to activate this behavior
(thanks to Jamie Cansdale for the tip).

 

Bug discovered in covered code = Easier fix

When a bug is discovered
in a portion of code, fixing it is easier if the potion of code is already
covered by tests. Indeed, if the buggy code is already covered by at least one
test, often one just needs to copy/paste this test and tweak its input to
reproduce the conditions where the bug appear. This extra test is then useful
both for:

  • Reproducing the bug at
    whim, to understand its conditions and fix it
  • Keeping verification that
    in the future will check that the bug won’t re-appear.

Having high test coverage
ratio increases the chance that most of the bug reside actually in code
covered. Ultimately, 100% test coverage means that if the code contains some
bugs, they are necessarily covered anyway by some tests.

 

High coverage leads to relevant investigation

Numerous times, while climbing the full coverage mountain, I
stumbled on dead code or better said dead condition. Imagine that in the code
below, despite plenty of tests the return
statement is never covered, meaning that obj
is never InAParticularState.

If(obj.IsInAParticularState) { return; }

It is a good hint to
investigate. Very often in such situation, the investigation will lead to the
fact that for some reasons, obj
cannot be InAParticularState at that
point. The best thing to do is then to turn the test condition into a contract.
This contract asserts that obj cannot
indeed be InAParticularState.

Debug.Assert(!obj.IsInAParticularState, 
“document the reason why object cannot be InAParticularState here”);

As a result:

  • you eliminated dead code,

  • the contract is covered
    by tests (and then verified at test time),

  • the contract constitutes
    a documentation for future developers for something that was not obvious at
    first glance and that required investigations.

 

100% Test Coverage protects from Coverage
Erosion

100% coverage ratio doesn’t
happen by chance. Writing a few tests can often cover up to 80% of the tested
code. However developers often have to struggle hard to reach the 100% coverage
goal. By struggling hard, I mean that developers need to write non-trivial tests
to cover non-mainstream scenarios. The 80/20 rule regularly applies here: these
extra 20% of code to cover can require up to 80% of the time spent in writing
tests. So when a class or a namespace is 100% covered it means one thing:
developers worked hard to reach this 100% value.

 

100% coverage then
becomes a requirement for future evolutions and refactoring: because the code
refactored is originally 100% covered, one has to make sure that the new
version of the code is also 100% covered. In other words, when code is 100%
covered it is easier to prevent what I usually name coverage erosion. The phenomenon of coverage erosion happens when
code gets refactored with poor care for writing new tests. Unfortunately, in
real-world development shop where turn-over and urgent evolutions are the
rules, coverage erosion is often a reality. It is what Jeff Atwood presented as
The theory of the broken window. Basically

  • if one develops in a
    clean environment, then one will struggle to keep the environment clean and
    avoid erosion.

  • if one develops in a
    dirty environment, then one won’t even try to make the environment cleaner and the
    overall entropy will increase.


This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Bill Schneider

    Revealing dead/unnecessary code is one of the best side effects of coverage.   Also it helps encourage you to refactor, since well-designed code with proper separation of concerns is easier to cover with unit tests.  

    http://wrschneider.blogspot.com/2012/01/unit-test-coverage-unintended-benefit.html  

    It’s also a good point that code that, if you already have test coverage, it’s easy to add assertions later when bugs are found.  At that point, you’re better informed about what assertions are useful.

  • http://andriybuday.blogspot.com Andriy Buday

    Very interesting and USEFUL article. These days I’m doing refactoring of code covered well by UT, and it is very important to keep an eye on having new tests. Your article helps me understand this better. Thanks.

  • http://weblogs.asp.net/gunnarpeipman/ Gunnar

    Nice to see Patrick you are also using code contratcs. I have special contracts page in my blob http://weblogs.asp.net/gunnarpeipman/pages/code-contracts.aspx where you can find my postings, presentation and example solution. Dunno, may be you can find something useful to you.

  • http://alexandregazola.wordpress.com Alexandre Gazola

    I think ‘good test coverage’ is a good thing… though it’s important to assess the real ‘quality’ of the written tests… but I do think they can be correlated.

  • http://rafanoronha.net/ Rafael Noronha

    Patrick,

    I think when high test coverage becomes illusion, maybe there is a problem in the quality of the code or the quality of the tests.

    In this case code coverage should not be the major issue.

  • Tormod

    The term “quality” is subjective. High test coverage is most certainly an assett, but it is still possible that the code does something that it shouldn’t or that it does not do something that it should.

    There is always a chance that you’ve missed some bounds check or somesuch that will result in some edge case, or just that you haven’t tested write permissions on your mocked disk or some other exception that may arise.

    100% test coverage will help you attack the problem when new bugs are discovered. You just need to add a new case add possibly add the new behaviour to your mocks to reproduce them.

    When you code the 100% civered code base you will be playing for keeps. You will be checking in better code than you are checking out. You will be working incrementally. And in many books this is code quality.
    You can still have a lousy application.

  • http://codebetter.com/members/Patrick-Smacchia/default.aspx Patrick Smacchia

    Alexander, good remarks indeed. The synthesis of what your said is that full code coverage can help handling ‘syntax/versionning’ problems that cannot be handled by the compiler.

    Jack, the idea of DbC is that verification are done by side-effect free code, meaning verification code don’t change any state, meaning whether contract verification are here or not the code behavior is exactly the same.

    debug.assert MUST be used with this ‘side-effect free code’ in mind. Your behavior delta between debug/release version is not due to debug.assert but the bad usage of it, something like:
    Debug.Assert(obj.ChangeState());

  • http://jack-fx.com jack

    I don’t like debug.assert, since it may result in some problem that only happens in release version.

  • http://nuald.blogspot.com Alexander Slesarev

    I want to notice few other advantages of using full code coverage:
    – useful for dynamic programming language projects or projects that use dynamic generation of code. Tests can check method’s argument types and help to avoid situation when parameters with incorrect types are passed to a method.
    - full code coverage can help with code analysis. For example, certain frameworks give warnings (like deprecated method call) only during runtime (like Qt, Gtk etc). We can catch all these warnings in the testing phase.
    - compatibility testing. For example, I can test my project in python2.4 and python2.6. Full code coverage give me guarantee that the project will work fine for the both environment.

  • http://codebetter.com/members/Patrick-Smacchia/default.aspx Patrick Smacchia

    Chris, I just did the test with NCover v3 and get a nice 100% coverage. With Reflector or NCoverExplorer I can see indeed the method generated uncovered but the fact is that these methods doesn’t have any associated PDB symbols. And coverage tools only count coverage of PDB symbols, hence the 100% coverage number ontained.

  • http://www.tavaresstudios.com Chris Tavares

    If ‘yield return’ is the source of perf issues in my code, I’ll be pretty darn happy. :-)

    Just write anything that looks like this:

    class Foo
    {
    public IEnumerable GetFoos()
    {
    yield return 1;
    yield return 2;
    yield return 42;
    }
    }

    Foo f = new Foo();
    foreach(int i in f.GetFoos())
    {
    Console.WriteLine(i);
    }

    And look at the all up coverage numbers. You’ll see that inner class the compiler generates for the yield statement has I think three methods on it at 0% coverage. I’ve seen this with both VS and NCover coverage.

    The only way to get them covered would be to explicitly get the enumerator and call the methods on it, instead of just putting it in a foreach loop. But at that point, you’re just writing test code to get coverage, not testing anything that’ll actually be called.

  • http://codebetter.com/members/Patrick-Smacchia/default.aspx Patrick Smacchia

    Chris, I am not a fan of ‘yield return’ syntax, mainly because of underlying perf reasons. But I didn’t know this fact, I am very surprised. Do you have a clean example to submit? Are you using VS or NCover as a coverage technology?

  • http://www.tavaresstudios.com Chris Tavares

    Unfortunately, true 100% coverage is impossible the second you use a “yield return” statement. The underlying iterator classes generate extra methods to match the IEnumerable interface that’ll never be called.