Normal 0 21 false false false FR X-NONE X-NONE

I heard several times that high test coverage ratio, and ultimately 100% test coverage, is an illusion of quality. The underlying reason is that when a portion of code is executed by tests, it doesn’t mean that the validity of the results produced by this portion of code is verified by tests. In other words, some code can be covered by tests and still contains bugs.

I totally agree with this assertion. However, I don’t agree when one claims that high test coverage ratio is an illusion of quality. I can see several major benefits resulting from high test coverage ratio.

Contract + High Test Coverage Ratio = Correctness checked anyway

If you apply correctly Design by Contract (DbC) high test coverage pays off. DbC means that your code contains side-effect free code whose only purpose is to check the states correctness at runtime as often as possible. If the state correctness is violated, a contract fails abruptly. To implement the DbC principle, we (the team working on NDepend) used so far my good old friend System.Diagnostics.Debug.Assert(…) and we are in the process of switching to the System.Contract API.

The trick is that if contract validity verifications are performed during tests execution, then tests are failing when a contract is broken. And the higher the test coverage ratio, the more contracts are verified at test-time. By extension, one can actually consider test code dedicated to verification (I mean lines in your test like Assert.IsTrue(…)) as contracts residing outside the code itself. Also, on NDepend development, we observe that regression bugs are discovered much more often thanks to broken contracts, than thanks to broken test. It is because contracts actually live in the code, they are much more close to what code is doing than test themselves. Also contracts are more often verified than tests (imagine a contract in a loop vs. a test verification outside the loop).

One technical detail is that TestDriven.NET discards Debug.Assert(…) failure windows by default and I explain here how to activate this behavior (thanks to Jamie Cansdale for the tip).

Bug discovered in covered code = Easier fix

When a bug is discovered in a portion of code, fixing it is easier if the potion of code is already covered by tests. Indeed, if the buggy code is already covered by at least one test, often one just needs to copy/paste this test and tweak its input to reproduce the conditions where the bug appear. This extra test is then useful both for:

• Reproducing the bug at whim, to understand its conditions and fix it
• Keeping verification that in the future will check that the bug won’t re-appear.

Having high test coverage ratio increases the chance that most of the bug reside actually in code covered. Ultimately, 100% test coverage means that if the code contains some bugs, they are necessarily covered anyway by some tests.

High coverage leads to relevant investigation

Numerous times, while climbing the full coverage mountain, I stumbled on dead code or better said dead condition. Imagine that in the code below, despite plenty of tests the return statement is never covered, meaning that obj is never InAParticularState.

If(obj.IsInAParticularState) { return; }

It is a good hint to investigate. Very often in such situation, the investigation will lead to the fact that for some reasons, obj cannot be InAParticularState at that point. The best thing to do is then to turn the test condition into a contract. This contract asserts that obj cannot indeed be InAParticularState.

Debug.Assert(!obj.IsInAParticularState, 
“document the reason why object cannot be InAParticularState here”);

As a result:

• you eliminated dead code,

• the contract is covered by tests (and then verified at test time),

• the contract constitutes a documentation for future developers for something that was not obvious at first glance and that required investigations.

100% Test Coverage protects from Coverage Erosion

100% coverage ratio doesn’t happen by chance. Writing a few tests can often cover up to 80% of the tested code. However developers often have to struggle hard to reach the 100% coverage goal. By struggling hard, I mean that developers need to write non-trivial tests to cover non-mainstream scenarios. The 80/20 rule regularly applies here: these extra 20% of code to cover can require up to 80% of the time spent in writing tests. So when a class or a namespace is 100% covered it means one thing: developers worked hard to reach this 100% value.

100% coverage then becomes a requirement for future evolutions and refactoring: because the code refactored is originally 100% covered, one has to make sure that the new version of the code is also 100% covered. In other words, when code is 100% covered it is easier to prevent what I usually name coverage erosion. The phenomenon of coverage erosion happens when code gets refactored with poor care for writing new tests. Unfortunately, in real-world development shop where turn-over and urgent evolutions are the rules, coverage erosion is often a reality. It is what Jeff Atwood presented as The theory of the broken window. Basically

• if one develops in a clean environment, then one will struggle to keep the environment clean and avoid erosion.

• if one develops in a dirty environment, then one won’t even try to make the environment cleaner and the overall entropy will increase.