Code Coverage is an oft-used metric which helps measure the depth and completeness of your testing. You run tests, and a code coverage tool monitors the execution and tracks which lines of code have and haven't been executed. You then get a report which breaks down the test coverage offered by your tests.
Here's an example coverage report generated from NCover:
(This is an old and free version of NCover, when will Resharper include Code Coverage?!)
On the left-hand side, we get an overview per assembly, namespace, type and method. You can see that my coverage isn't great. On the right, you see the actual code. Notice that the instantiation of my InternalCategories isn't covered by any tests (the light-red background).
Code coverage is a useful tool, but not quite as useful as it might first seem. The fundamental problem with code coverage, and something you must always remember when using it, is that it does not measure meaningful execution. Take a look at the following method and test:
public void HasChanged(User original)
{
return string.Compare(original.Name, Name, true) != 0
|| string.Compare(original.Password, Password, true) != 0;
}
[Test]
public void Returns_False_If_Nothing_Has_Changed()
{
var updated = new User{Name = "Name", Password = "Password"};
var original = new User{Name = "Name", Password = "Password"};
Assert.AreEqual(false, original.HasChanged(updated));
}
Running the above with code coverage will report that our code is 100% covered even though we've only tested part of the actual functionality. This is both misleading and dangerous. We might be tempted to just add another test:
[Test]
Public void Returns_True_If_Name_Has_Changed()
{
var updated = new User{Name = "Gamblor", Password = "Password"};
var original = new User{Name = "Name", Password = "Password"};
Assert.AreEqual(true, original.HasChanged(updated));
}
But we still have to add a third test to get full coverage:
[Test]
Public void Returns_True_If_Password_Has_Changed()
{
var updated = new User{Name = "Name", Password = "Password"};
var original = new User{Name = "Name", Password = "drowssaP"};
Assert.AreEqual(true, original.HasChanged(updated));
}
(some might argue that we need even more since none of these test handle the case-sensitive behavior of or original method.)
The point is simple: code coverage cannot be used exclusively as a reliable means to measure the quality or even the coverage or your tests. On the flip side though, you can use coverage to identify code that isn't tested. In other words, just because a method or line is covered doesn't mean it's tested, but if a line or method isn't covered, you can be absolutely sure that it isn't tested.
Posted
12-09-2008 9:26 AM
by
karl