Code Coverage: What Is It Good For?

Absolutely nothing. Or so the story goes…

I had a mini-exchange with my pal Glenn Block last night on Twitter. He was recycling the somewhat old meme that code coverage is a cockamamie metric that, as he puts it, ” a false security blanket on quality”.

Measurements (other than net profit, of course) are merely indicators. They give us insight into what we’re doing, red flags that help us know when to follow up on and drill into with more qualitative and hands on tools.This is, more-or-less, the idea behind Autonomation or “automation with a human touch.”

For example: while code coverage isn’t proof that our testing efforts will yield higher maintainability, it does tell us a teams commitment to a test practice.

I tend to believe the most interesting indicators are composite metrics. What happens when we start thinking about combining code coverage with cyclomatic complexity? I’d like to read a (hopefully) inverse relationship over time, coverage rising while complexity falls, as a good indicator that we’re doing simple design.

The important thing to remember about metrics is they are only single purpose tools which, correctly applied, gift us with little tidbits of insight. It’s up to you to find meaningful and creative uses for the various metrics and their various combinations. Once you start doing that, measurements like code coverage can certainly help you achieve your immediate goals.

This entry was posted in Metrics, practices, TDD, teams, testing. Bookmark the permalink. Follow any comments here with the RSS feed for this post.

10 Responses to Code Coverage: What Is It Good For?

  1. Jesse Gibbs says:

    I second the comment by a few previous readers that code coverage tools can reveal some important insights, including:

    * The ‘riskiest’ areas in your codebase: A combination of low coverage and high compexity definitely warrants investigation.

    * Loss of code coverage over time is an indicator that code is changing but tests are not, which can be problematic.

    IMHO, much of the negativity towards code coverage metrics is due to their misuse. The statistics can easily be used to instill a false sense of security in stakeholders who don’t understand that touching a line of code w/ a test doesn’t make it bug-proof. I’d prefer to see code coverage metrics as a tool used within the development team to prioritise where to invest in new and updated unit and functional tests.

  2. Hi Dave,

    We do a weekly webcast on testing called “This week in testing” (http://learn.typemock.com/this-week-in-test/2009/11/18/episode-4-animals-that-talk-back.html)
    and we’ve mentioned this post (and the one about the cost of losing a developer).

    Please come and watch, and if you like what you see, please tell other people as well.

    Thanks,

    Gil Zilberfeld
    Typemock

  3. Rob says:

    Composite metrics are definitely the way to go. The more different ways you are able to measure your code quality the better. I’ve been playing with integrating NDepend and NCover into Teamcity and have come up with some nice visual dashboards:

    http://blog.robbowley.net/2009/10/28/visualising-the-internal-quality-of-software-part-1/

  4. Nick says:

    To be honest, you really have to look at functional coverage rather than actual coverage made by writing tests.

    People can (wrongly) engineer tests that will check down different branching points in code, but don’t actually identify the fact that something should or shouldn’t happen.

    This is where imho, BDD is a good route to take, as it describes the functionality of a piece of code in the given a set of circumstances.

  5. Dan Pickett says:

    Like anything, I think metrics are best used and studied and moderation. When coming on to a new project, you can glean a lot of insight just by getting code coverage. It helps to diagnose problems, not resolve them.

    Some people I know are perfectionists when it comes to this metric. 100% code coverage in and of itself is kind of a myth – you still have integration issues as well as potential false positives with mocks and stubs. You described it really well when you said it can act like a “security blanket”

  6. Steve Py says:

    I don’t put much importance on the coverage “metric” but looking at code coverage as a whole is certainly important. Identifying areas that aren’t covered is critically important, but also the number of times each element is “touched” can be equally important especially when CC is unavoidable. The metric may say that nearly every line is touched, but that certainly lends nothing to assessing whether each combination has some form of coverage.

    Bad tests can spoof any effort to mine information out of coverage so it’s certainly *not* enough to just say “The code is safe, the code is clean, so long as those bars are mostly green.” Personally I review as much of the code, and unit tests as time affords me.

  7. mendicant says:

    Hearing about this the last couple days I realized that it’s not really the coverage number itself that I usually pay attention to but rather what it’s doing. When it’s falling I wonder what’s happening. When it’s rising I feel a bit more comfortable.

    It also helps that I fully trust the testing capabilities of my coworkers.

  8. Dax says:

    So the question is, how does one convince the FDA?

  9. Right, it’s a metric. Complaining about the information it gives you is like complaining about temperature not giving you the position and velocity of all molecules. That’s the point of the metric: to reduce the view on a system to a single value. It does not characterize the system, it measures something. Only by adding other metrics can you get a more real idea of the underlying system. The difficulty is to know what and how much you need to know.

  10. Rob says:

    Agreed. Code metrics merely open your eyes to what you are doing and not doing. It’s then the developers’ job to fix what is important.

    That is the metrics only value: Insight.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>