Sponsored By Aspose - File Format APIs for .NET

Aspose are the market leader of .NET APIs for file business formats – natively work with DOCX, XLSX, PPT, PDF, MSG, MPP, images formats and many more!

Code Coverage : Use It Wisely

Code Coverage is an oft-used metric which helps measure the depth and completeness of your testing. You run tests, and a code coverage tool monitors the execution and tracks which lines of code have and haven’t been executed. You then get a report which breaks down the test coverage offered by your tests.


Here’s an example coverage report generated from NCover:


(This is an old and free version of NCover, when will Resharper include Code Coverage?!)


On the left-hand side, we get an overview per assembly, namespace, type and method. You can see that my coverage isn’t great. On the right, you see the actual code. Notice that the instantiation of my InternalCategories isn’t covered by any tests (the light-red background).


Code coverage is a useful tool, but not quite as useful as it might first seem. The fundamental problem with code coverage, and something you must always remember when using it, is that it does not measure meaningful execution. Take a look at the following method and test:

public void HasChanged(User original)
{
return string.Compare(original.Name, Name, true) != 0
|| string.Compare(original.Password, Password, true) != 0;
}

[Test]
public void Returns_False_If_Nothing_Has_Changed()
{
var updated = new User{Name = “Name”, Password = “Password”};
var original = new User{Name = “Name”, Password = “Password”};
Assert.AreEqual(false, original.HasChanged(updated));
}


Running the above with code coverage will report that our code is 100% covered even though we’ve only tested part of the actual functionality. This is both misleading and dangerous. We might be tempted to just add another test:

[Test]
Public void Returns_True_If_Name_Has_Changed()
{
var updated = new User{Name = “Gamblor”, Password = “Password”};
var original = new User{Name = “Name”, Password = “Password”};
Assert.AreEqual(true, original.HasChanged(updated));
}

But we still have to add a third test to get full coverage:

[Test]
Public void Returns_True_If_Password_Has_Changed()
{
var updated = new User{Name = “Name”, Password = “Password”};
var original = new User{Name = “Name”, Password = “drowssaP”};
Assert.AreEqual(true, original.HasChanged(updated));
}

(some might argue that we need even more since none of these test handle the case-sensitive behavior of or original method.)


The point is simple: code coverage cannot be used exclusively as a reliable means to measure the quality or even the coverage or your tests. On the flip side though, you can use coverage to identify code that isn’t tested. In other words, just because a method or line is covered doesn’t mean it’s tested, but if a line or method isn’t covered, you can be absolutely sure that it isn’t tested.

This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.

14 Responses to Code Coverage : Use It Wisely

  1. Stuart Dootson says:

    Rather than the term ‘code coverage’, you probably want to be saying ‘statement coverage’. This then makes it obvious that there are many other sorts of coverage – if NCover measured and showed you branch and modified condition/decision coverage metrics, you’d get an indication that the different paths through your code weren’t covered, even though all statements were covered.

    It isn’t rocket science – the safety-critical world has used these metrics for decades….

  2. The traditional understanding of code coverage has been exactly what you just wrote about — ONLY a NUMBER. In some esoteric sense, you were supposed to decide if that was good, bad, or acceptable (or who gives a sh*t the boss is asking for it)…

    For the first time in code coverage, NCOVER 3 will attempt to provide “MEANING” behind the numbers…

    Symbol points, method visits, branch points, cyclomatic complexity — all trended and graphed to show you a deeper meaning.

    Get it ALL for FREE right now… email us at: conversation [at] ncover [dot] com

  3. Grant Palin says:

    I wrote about code coverage a while ago, and I had reached much the same conclusion: coverage isn’t everything. The way I look at it: coverage is part of the process, not the end result.

    When I look at my coverage numbers, I use the data to learn what is NOT currently under test, as opposed to assuming that full coverage means the code is good.

    As I see it, coverage is just one tool, along with others like FxCop and NDepend. And of course, the occasional manual code review.

  4. Jerry says:

    @brian

    the note re: meaningful execution was from the blog entry directly – in general I was agreeing with you in general, just bringing in a note from the original entry to illustrate my point.

  5. Brian says:

    @Jerry

    [quote]
    but, to state that code coverage in general does not test meaningful execution, when one single tool does not, is just plain silly, painting a broad stroke with a generalization.
    [quote]

    That’s not what I was trying to say; I was merely pointing out that those with less experience (such as myself at one point), might be misled to believe that ‘green’ = done.

    Also, when did I generalize that there was nothing meaningful in code coverage in general? I never meant to imply that there wasn’t anything meaningful in code coverage – in fact it’s the first thing I look at after writing tests to ensure what I ‘think’ covered all the test cases actually at least hit all the code I thought it would.

  6. karl says:

    Jerry:
    Your critique is spot on. I’ve used a few and I’m blogging based on my experience with those. Thanks for letting me (and readers of this blog) know about tools which make my assumption less accurate. Definitely gonna look into it.

  7. Jerry says:

    @Brian

    just because it’s green doesn’t mean 100% of TEST CASES are covered

    there will never truly be 100% test coverage – nor is it feasible.

    but, to state that code coverage in general does not test meaningful execution, when one single tool does not, is just plain silly, painting a broad stroke with a generalization.

  8. Brian says:

    The code coverage tool that comes with Visual Studio works pretty well at getting the extra ‘truth table’ rows – for instance it would of picked up what your 3rd party product says was 100% covered.

    But yes…for people who use code coverage tools and don’t already know this – they’re not silver bullets, they’re not guarantee’s as everyone else has mentioned (or hinted at) it’s excellent for telling you where you have no tests (or little tests – Visual Studio highlights those as yellow when you only have partial coverage) – but just because it’s green doesn’t mean 100% of TEST CASES are covered.

  9. Hi Karl,

    Nice post! I agree that code coverage is not the final measure for your application. Also, if your code coverage is 96% then this should not be taken as a positive measure but we have to think that what 5% code we are not covering.

    You can also hack code coverage:

    http://azamsharp.com/Posts/63_Hacking_Code_Coverage.aspx

  10. Colin Jack says:

    Nice example, I usually see low overall coverage as a sign of a problem but high coverage does not give me confidence.

  11. Jerry says:

    You appear to be making a generalization based on a specific instance. While I agree that code coverage should never be relied on as the only tool in the toolbox, some code coverage tools do a good job of checking every truth value and not marking something as covered until each and every truth value has been determined.

    See for example:

    http://search.cpan.org/~pjcj/Devel-Cover-0.64/lib/Devel/Cover.pm

  12. Itay Maman says:

    Here’s a real story. We worked on a project that was pretty well tested. Still, one day we discovered that one of the getter methods fails with a null-pointer-exception if its instance was created via the default (no-arg) constructor. It is a something that our tests didn’t cover.

    One of our team members decided to overcome this lack of testing (as he called it) by writing a test that goes over all classes in our project, instantiates each one and invokes every getter method. This, of course, made sure that the problem will not repeat it self.

    This approach is not as good as it sounds. Although it raised the coverage of most getter methods to 100% it did not check the return values. We were no longer able to use coverage to detect the methods that had no real test.

    Eventually, we removed this artificial test from the suite.

  13. José Tavares says:

    Nice article explaining the problems with code coverage.
    It would be nice though if at least mention other coverage criteria like decision and coverage criteria.

  14. Jason says:

    Nice explanation. I’ve always considered code coverage as 0 or X measure (as opposed to 0 or 1) – either you know for sure you have NOT covered a block, or you know you’ve covered something, but the raw numbers alone can’t give you enough confidence to say it’s tested.