Full Coverage by Tests Driven Development

Posted by Patrick Smacchia on July 13, 2010

 

I wrote in a past post about my passion for code 100% covered by tests. High
Test Coverage Ratio is a good thing, Anyway!

To summarize the main motivations exposed in the
post are:

  • Using code contract coupled with high coverage ratio leads to
    checked correctness.
  • A bug discovered in tested code is easier to fix.
  • High test coverage guides mechanically the developer to potential
    flaws in the code.
  • Code fully covered by test is less subject to code erosion.

An essential note is that whatever the API used to
implement contract (.NET 4 Code Contract, Debug.Assert(…)…)
automatic tests shouldn’t provoke a contract violation. A contract
violation means buggy code. If a test execution violates a contract
either the code or the test or both should be fixed. The idea is that
when practicing full coverage by tests driven development
contracts are utterly important for checking correctness that is not
necessarily checked by tests.

Today, one year after having written this post, I
realize that I am even more extremist concerning full code coverage
by tests coupled with code contract. I attest that the extra-cost of covering all
branches of my code + writing as many contracts as I can is well
balanced by the time and effort saved. A lot of development resource is
saved thanks to the avalanche of flaws discovered early at development
time. I attest that the number of bugs discovered afterthought in
such code is, dare I say it, null! After having written set of
collaborative classes 100% covered by tests I feel the great impression
of a job done, what’s next?

Unfortunately, not all code is covereable by tests
and I wrote on this topic in the past: Dealing
with Code Un-Coverable by Tests
In this post, I explained a solution we used with NDepend to both
segregate and tag code uncoverable by tests.

Making sure that 100%
test coverage ratio won’t decrease in the future

Actually we have a cool trick to keep
coverage ratio to its maximum. In the assembly NDepend.CQL.dll
there is a public attribute class named FullCoveredAttribute.
This attribute can tags assembly, class, struct, method and
constructor. We use this attribute mostly to tag classes 100% covered by
automatic tests. Under NCoverExplorer this looks like:

 

We did half the job here, to develop a class
100% covered by tests with contracts. But the other half of the job is
to make sure that future changes in the class will be also covered. This
is where the FullCovered attribute comes into play. Code Query
Language (CQL) can deal both with attribute tagging and
code coverage ratio metrics. Hence, it is really easy to write a CQL
rule to check that classes tagged with the attribute FullCovered
is also 100% covered by tests:

WARN IF Count > 0 IN
SELECT TYPES
WHERE HasAttribute
“NDepend.CQL.FullCoveredAttribute“
 AND
PercentageCoverage <
100

I can attest that this
rule warns regularly, reminding the team that some classes has been
refactored and that some new code hasn’t been properly covered. Clearly,
having the FullCovered attribute tagging the class declaration
is not a high enough indication for developers to care for full
coverage. An automatic verification process must takes place through the
execution of the CQL rule.

Notice that nothing prevents you from having your own
attribute class instead of NDepend.CQL.FullCoveredAttribute.

 

As a final note, concerning coverage technology we are
happy users of NCover coupled with TestDriven.NET
for VS integration and NUnit as
a test Fx. What I appreciate is the awesomely tiny overhead of NCover
on code execution and the numerous convenient tricks of TestDriven.NET!

Share/Bookmark

This entry was posted in Full Coverage, Tests Driven Development. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://codebetter.com/members/Patrick-Smacchia/default.aspx Patrick Smacchia

    >Fair enough, although strange argument coming from someone depends exclusively  on numbers and measures.

    DaRage,I believe indeed that software engineering needs more measures and numbers and more clean code with better componentization and higher mesurable quality. Doing so has been proven to be useful in all of other engineering activity for centuries. I don’t rely exclusively on numbers, as in other engineering activities, talented and experienced engineers can make a non-quantifiable positive difference.

     

    >So based on your experience, what would be the balance of effort vs time saved if the code coverage was 80%?

    The balance effort vs time depends on many parameters, especially the number of years the project will be maintained and the number of time the code will face refactoring because of feature evolutions.

    Concerning NDepend which is a project I know pretty well, I can say that having a lot of 100% covered components has saved us a lot of frustrations and time, quantifiable in N months per year, with N increasing each year. If you look at our release notes, you’ll see that the project never stop to move on, and I attest the this sustainable rhythm comes from a code base with a small technical debt.

    http://www.ndepend.com/ReleaseNotes.aspx


    >Is unit testing  a special case and thus %100 coverage is needed or does the 80-20 rule apply and we can find 80% of issues with only  %20 coverage? Or is the answer somewhere in between? this is what I’m trying to find.

    Indeed 80% of the time is spend in the last 20% of lines of code to cover. But interestingly enough, I’d say that 80% of the unexpected issues found are also found in these 20% of line to cover so it is worth the effort. IMHO the underlying reason is simple: if a portion of code is hard to cover by tests it often mean that the semantic of this portion of code is complex, it represents edge-cases often neglected by developers, hence it is error prone.

  • http://codebetter.com/members/Patrick-Smacchia/default.aspx Patrick Smacchia

    >What about code that you can’t 100% cover?

    Chris, I explained all my positions in a previous blog post linked in the article:
    http://codebetter.com/blogs/patricksmacchia/archive/2008/07/22/dealing-with-code-un-coverable-by-tests.aspx

  • http://codebetter.com/members/Patrick-Smacchia/default.aspx Patrick Smacchia

    Rainer, if you’re in a software commercial shop that cannot afford a few hundreds bucks on tooling for developer my recommendation is to look for another job somewhere else.

    If you’re student, academic searcher or working on OSS projects, my recommendation is to ask for a free license in exchange of a bit of promotion (like an online article about the tool, a ‘powered by NCover’ tag on your website etc…)

  • http://www.py-sty.blogspot.com Steve Py

    @DaRage

    I’d say that 100% coverage for the sake of having full coverage is excessive. In my view, 100% of business logic in an application must be tested. Adopting aenemic views, and recent explorations into MVVM means the important code lives in fully testable regions of the application and as a result many classes have 100% test coverage.

    TDD is an important topic to consider when looking at test coverage solely because adopting a test-driven approach you are designing code with testability being a primary consideration. If you’re writing code solely to satisfy requirements and then thinking about if and how it should be tested afterwards, you’re facing a more difficult task to get high test coverage; This makes it all that much harder to justify the effort.

  • DaRage

    @Steve

    The topic of the discussion is not TDD but full coverage. Do you think that working toward %100 coverage is worthwhile and justifiable? Patrick is arguing for. I think that %100 of anything is an extreme, by definition, and raises the question whether it stems from a real necessity or an obsession with statistics.

    Is unit testing a special case and thus %100 coverage is needed or does the 80-20 rule apply and we can find 80% of issues with only %20 coverage? Or is the answer somewhere in between? this is what I’m trying to find.

  • http://www.py-sty.blogspot.com Steve Py

    >how could you know how much time you saved when didn’t have any bugs?

    As Patrick pointed out with experience this becomes easier to estimate. As a contractor I unfortunately still find myself on far more projects without any structured testing discipline than those that do. I advocate for investing in test focused development approaches but it’s one voice against the choir of bias. I see first-hand the costs involved when projects have little to no repeatable tests. Techniques like TDD and TSD don’t add much if anything to the cost of a project. Any developer worth anything will try out the code they write and “make sure it works.” The difference with using a more disciplined approach with automated unit tests is that you invest a healthy amount of that checking time as repeatable tests up-front to avoid a volatile quantity of fixing time afterwards. What you gain is significant regression issue detection, something that developers rarely test for “as they go.”
    http://py-sty.blogspot.com/2010/01/cost-of-tdd-test-soon-development.html

  • http://www.tavaresstudios.com Chris Tavares

    What about code that you can’t 100% cover? For example, if you use a C# yield statement, you get a full IEnumerator implementation, including several methods that will never get called. Unless you explicitly go and cast the iterator into an IEnumerator and call those methods only for a test. And then you’ve just wasted your time, because not only are those methods never called in your code, they’re generated by the compiler anyway and why test the compiler?

  • DaRage

    @Patrick

    Fair enough, although strange argument coming from someone depends exclusively on numbers and measures. So based on your experience, what would be the balance of effort vs time saved if the code coverage was 80%?

  • Rainer Hilmer

    What’s your recommendation for people who don’t or can’t have NCover for whatever reason?

  • http://codebetter.com/members/Patrick-Smacchia/default.aspx Patrick Smacchia

    > how could you know how much time you saved when didn’t have any bugs?

    Good remark!
    I am programming professionally for 15 years and I began coding when I was 10. In my programming life I spent countless time fixing bugs of other programmers. This gives me a pretty good estimation of the cost of bugs discovered late or even at production time.

  • DaRage

    You say that the time spent was well balanced by the effort saved but how could you know how much time you saved when didn’t have any bugs?