Code UnCoverable by Tests
Even for test-coverage
addict (as me), there is some code that simply cannot be covered by tests. An
example?
The call to MessageBox.Show() cannot be tested
automatically since it is blocking. Of course we could mock calls to MessageBox.Show(), but at the end of
the day, there will be at least one call to this method in the code base, one
call that cannot be covered by tests.
This example is not
isolated. There are a multitude of cases where code reacts to things that
cannot be automatized, such as exception that cannot be reproduced…
…but also some environment
settings, folder or file browsing by user, showing a modal custom dialog and
more.
The Problem
As we explained above, while
using mocking can help reduce the surface of code not-coverable by tests, it can’t
help to get a code base 100% covered. When exploring test coverage results,
with NCoverExplorer, Visual Studio Team System
or NDepend, this problem will lead
to classes 98% or so covered. It is taking time and it is error prone to figured out each time you are exploring coverage results, that
these are false alert. This is clearly anti-agile! What we want is a clean 100%
coverage result!
The Solution
NCover and NDepend are tackling this problem the same way: you can exclude some methods from coverage results through a dedicated
(and eventually custom) attribute.
NCover knows
about this attribute through the /ea command
line option as describe here
by Jamie Cansdale. Code tagged with
this attribute will be excluded from coverage statistics and won’t disturb while
exploring test coverage results. For example, if all methods of the class Foo are 100% covered except the method MessageBoxShow(), if the method MessageBoxShow() is tagged with the
coverage exclude attribute then the class Foo will be shown as 100% covered. A great side-effect is that developers reviewing the code will know about this coverage exclusion.
With NDepend, the coverage excluding attribute can be provided through
the import coverage file dialog as shown below. This will have the same effect
as with NCover. For convenience, the
assembly NDepend.CQL.dll can be linked
from your project. This assembly contains the dedicated attribute: NDepend.CQL.UncoverableByTestAttribute.
If you prefer you can provide your own attribute.
As far as I know, Visual Studio team System Coverage doesn’t
support yet this feature but it might be considered for future releases, as
explained here.
The Bonus
Actually the burden of exploring
coverage results to make sure that some classes or namespaces should be 100% covered
is not really agile. It is manual, time consuming, error-prone and cannot be
capitalized for future iterations.
NDepend can help automate
this task with just 2 CQL rules:
// <Name>Types 100% covered by tests</Name>
WARN IF Count > 0 IN SELECT TYPES WHERE
HasAttribute "NDepend.CQL.FullCoveredAttribute" AND PercentageCoverage < 100
// <Name>Methods 100% covered by tests</Name>
WARN IF Count > 0 IN SELECT METHODS WHERE
HasAttribute "NDepend.CQL.FullCoveredAttribute" AND PercentageCoverage < 100
Basically, these rules will
check automatically that all types and methods tagged with the attribute FullCoveredAttribute are indeed 100%
covered. Insert these rules in your build process and you’ll be informed as
soon as a lack of test coverage is luring. Here also, as a bonus side-effect
developer reviewing or refactoring the code will know instantly which part of the
code is supposed to be 100% covered. For applying this trick to the code base of NDepend since the code coverage feature is available, I can testify that it is a really, really time saving trick.
For convenience, the
assembly NDepend.CQL.dll can be linked
from your project and it contains the dedicated attribute: NDepend.CQL.FullCoveredAttribute (and also the attribute NDepend.CQL.MoreThan95PercentCoveredAttribute).
Of course, CQL rules can be readily tweaked to work with a custom attribute.