Lessons learned from the NUnit code base

I recently analyzed NUnit
v2.4.8 with NDepend.
The
first impression is that developers behind NUnit know their job and did
an
excellent work. This positive feedback comes from many details, the
fact that you can
seamlessly open the VisualStudio solution and compile everything
immediately, the amount of documentation, the high number of releases,
the number
of tests, the support for every version of .NET (and also non-MS ones),
IDE independent
code…

In terms of Lines of Code,
NUnit is relatively small with 12 559 LoC (not including tests assemblies).
To be compared to 70 681 LoC for NDepend v2.11,   
42 092 LoC for Paint.NET v3.22,   
36 143 LoC for NHibernate v2.0,   
29 791 LoC for Castle v2.0.3,   
474 388 LoC for Resharper v4.1,   
and around 2M LoC for the entire .NET Fx 3.5 (including WPF, WCF…).

Test Coverage of NUnit

  
In terms of code
coverage, NCover says
47% but some tests didn’t pass while running with TestDriven.NETereHere . Here is a
NDepend snapshot of methods more than 90% covered (rectangles in blue represent methods that are at least 90% covered by tests). Without surprise, UI code is poorly
covered.

Assemblies of NUnit: How to compile them 3 times faster?

While looking at
assemblies, an immediate remark comes to me: is there really a need for 19
assemblies for these 13 068 Lines of Code?

I wrote once about the benefits of having less but bigger assemblies.
One of the reason was the compilation time and indeed, the 13K LoC of NUnit +
4K Loc of test projects takes 32 seconds to compile (total of 29 projects). In
comparison the 70K LoC of NDepend are compiled in 11 seconds (total of 11 projects).
So how does such
a gap can be explained? 2 reasons, first the high number of
projects/assemblies and second, the use of the Copy Local project
reference option:

The VisualStudio Project Reference + Copy Local true option is evil! If you don’t believe me, go and check by yourself: because of this option the assembly
nunit.core.dll is duplicated 21 times
while compiling the NUnit solution with VisualStudio! It means that it has to
be copied/loaded/parsed 21 times by the C# compiler that apparently doesn’t come
with a cache/hashCode optimization for such a case.

So I did the experiment of
replacing projects references by assembly references. I also parametrized VisualStudio to output all assemblies in a
single folder (menu Project Properties
> Build > Output path). The
compilation times went from 32 seconds to 12 seconds! 

The second reason that
explains slow compilation is the high number of project (19 project + 10 test
projects). Clearly, if assemblies were merged into: few test assemblies, one GUI
assembly, one console assembly and one framework assembly, the compilation duration
could be decreased from 12 seconds to something like 3 seconds, a x10 factor compared to the
initial 32 seconds. Then a tool such as NDepend
could be used to guarantee that the structure of the code remain
properly layered.
And indeed, the code of NUnit is pretty well layered. This is the
result of the initial decision to create plenty of assemblies. This can
be shown by
looking at NUnit namespaces through a dependency matrix, there are only
2 small
cycles:

This is a great news compare
to the monolithic code structure of NHibernate for example, where all of the 62
namespaces literally depend on all others 62 namespaces (as described here).

 
Another thing to notice
is that this high number of assemblies forces NUnit consumer projects to reference
several assemblies and also to deploy all of them, without really knowing if
one is missing or not. It would be nice to just be able to reference nunit.framework.dll and be able to just
copy/paste it at whim.

Assemblies of NUnit: Insider view

By contacting Charlie Poole,
the lead developer of NUnit, to have its opinion before publishing the
current post, I had the surprise that Charlie spontaneously confesses
that reducing the number of assemblies is indeed part of their upcoming plans.
This also come with some justification for each assembly, here are Charlie’s remarks:

• Two
  assemblies, nunit.exe and nunit-console.exe do nothing but call the
  main of the two runner dlls. This is a feature requested by folks who
  want to run NUnit out of their own exe.
• The uikit assembly was made separate so that it could be used in alternative guis.
• The
  mock and framework assemblies are both referenced by users. But not
  many users use our mock framework, so they need to be able to reference
  the one they want to use instead.
• NUnit uses remoting, with
  some parts loaded in the test domain and others only in the primary
  domain. This impacts the minimum number of assemblies you want to have.
  Our split is currently between util and core, with core.interfaces
  pointing the way to a future split.
• NUnit has multiple runners,
  some we write, some other people write. So engine that runs tests has
  to be separate for those users.

NUnit Code Quality

Through the prism of classic metrics, the code quality is pretty neat: only 62 methods
on 3 233 slightly exceed classic thresholds (the report tells 4 388 methods because it includes tier code methods):

WARN IF Count > 0 IN SELECT METHODS WHERE !NameIs "InitializeComponent()" AND                                           // Metrics' definitions     (  NbLinesOfCode > 30 OR              // http://www.ndepend.com/Metrics.aspx#NbLinesOfCode        NbILInstructions > 200 OR          // http://www.ndepend.com/Metrics.aspx#NbILInstructions        CyclomaticComplexity > 20 OR       // http://www.ndepend.com/Metrics.aspx#CC        ILCyclomaticComplexity > 50 OR     // http://www.ndepend.com/Metrics.aspx#ILCC        ILNestingDepth > 4 OR              // http://www.ndepend.com/Metrics.aspx#ILNestingDepth        NbParameters > 5 OR                // http://www.ndepend.com/Metrics.aspx#NbParameters        NbVariables > 8  )                 // http://www.ndepend.com/Metrics.aspx#NbVariables        

NUnit Code Maintenance and Evolution
Finally
let’s also mention that the NUnit team is serious about maintenance and
evolution: more than half of the code base has been refactored between
v2.4.1 and v2.4.8. To read this metric view know that:

• Each rectangle represents a method
• The surface of a rectangle is proportional to the number of Lines of Code of the method
• Rectangles in blue are those matched by the CQL query below. In this context rectangles in blue are methods that have been refactored or added between
  v2.4.1 and v2.4.8.