I recently analyzed NUnit v2.4.8 with NDepend.
The first impression is that developers behind NUnit know their job and did an excellent work. This positive feedback comes from many details, the fact that you can seamlessly open the VisualStudio solution and compile everything immediately, the amount of documentation, the high number of releases, the number of tests, the support for every version of .NET (and also non-MS ones), IDE independent code…
In terms of Lines of Code, NUnit is relatively small with 12 559 LoC (not including tests assemblies). To be compared to 70 681 LoC for NDepend v2.11, 42 092 LoC for Paint.NET v3.22, 36 143 LoC for NHibernate v2.0, 29 791 LoC for Castle v2.0.3, 474 388 LoC for Resharper v4.1, and around 2M LoC for the entire .NET Fx 3.5 (including WPF, WCF…).
Test Coverage of NUnit
In terms of code coverage, NCover says 47% but some tests didn’t pass while running with TestDriven.NETereHere . Here is a NDepend snapshot of methods more than 90% covered (rectangles in blue represent methods that are at least 90% covered by tests). Without surprise, UI code is poorly covered.
Assemblies of NUnit: How to compile them 3 times faster?
While looking at assemblies, an immediate remark comes to me: is there really a need for 19 assemblies for these 13 068 Lines of Code?
I wrote once about the benefits of having less but bigger assemblies. One of the reason was the compilation time and indeed, the 13K LoC of NUnit + 4K Loc of test projects takes 32 seconds to compile (total of 29 projects). In comparison the 70K LoC of NDepend are compiled in 11 seconds (total of 11 projects). So how does such a gap can be explained? 2 reasons, first the high number of projects/assemblies and second, the use of the Copy Local project reference option:
The VisualStudio Project Reference + Copy Local true option is evil! If you don’t believe me, go and check by yourself: because of this option the assembly nunit.core.dll is duplicated 21 times while compiling the NUnit solution with VisualStudio! It means that it has to be copied/loaded/parsed 21 times by the C# compiler that apparently doesn’t come with a cache/hashCode optimization for such a case.
So I did the experiment of replacing projects references by assembly references. I also
parametrized VisualStudio to output all assemblies in a single folder (menu Project Properties > Build > Output path). The compilation times went from 32 seconds to 12 seconds!
The second reason that explains slow compilation is the high number of project (19 project + 10 test projects). Clearly, if assemblies were merged into: few test assemblies, one GUI assembly, one console assembly and one framework assembly, the compilation duration could be decreased from 12 seconds to something like 3 seconds, a x10 factor compared to the initial 32 seconds. Then a tool such as NDepend could be used to guarantee that the structure of the code remain properly layered. And indeed, the code of NUnit is pretty well layered. This is the result of the initial decision to create plenty of assemblies. This can be shown by looking at NUnit namespaces through a dependency matrix, there are only 2 small cycles:
This is a great news compare to the monolithic code structure of NHibernate for example, where all of the 62 namespaces literally depend on all others 62 namespaces (as described here).
Another thing to notice is that this high number of assemblies forces NUnit consumer projects to reference several assemblies and also to deploy all of them, without really knowing if one is missing or not. It would be nice to just be able to reference nunit.framework.dll and be able to just copy/paste it at whim.
Assemblies of NUnit: Insider view
By contacting Charlie Poole, the lead developer of NUnit, to have its opinion before publishing the current post, I had the surprise that Charlie spontaneously confesses that reducing the number of assemblies is indeed part of their upcoming plans. This also come with some justification for each assembly, here are Charlie’s remarks:
- Two assemblies, nunit.exe and nunit-console.exe do nothing but call the main of the two runner dlls. This is a feature requested by folks who want to run NUnit out of their own exe.
- The uikit assembly was made separate so that it could be used in alternative guis.
- The mock and framework assemblies are both referenced by users. But not many users use our mock framework, so they need to be able to reference the one they want to use instead.
- NUnit uses remoting, with some parts loaded in the test domain and others only in the primary domain. This impacts the minimum number of assemblies you want to have. Our split is currently between util and core, with core.interfaces pointing the way to a future split.
- NUnit has multiple runners, some we write, some other people write. So engine that runs tests has to be separate for those users.
NUnit Code Quality
Through the prism of classic metrics, the code quality is pretty neat: only 62 methods
on 3 233 slightly exceed classic thresholds (the report tells 4 388 methods because it includes tier code methods):
WARN IF Count > 0 IN SELECT METHODS WHERE !NameIs "InitializeComponent()" AND // Metrics' definitions ( NbLinesOfCode > 30 OR // http://www.ndepend.com/Metrics.aspx#NbLinesOfCode NbILInstructions > 200 OR // http://www.ndepend.com/Metrics.aspx#NbILInstructions CyclomaticComplexity > 20 OR // http://www.ndepend.com/Metrics.aspx#CC ILCyclomaticComplexity > 50 OR // http://www.ndepend.com/Metrics.aspx#ILCC ILNestingDepth > 4 OR // http://www.ndepend.com/Metrics.aspx#ILNestingDepth NbParameters > 5 OR // http://www.ndepend.com/Metrics.aspx#NbParameters NbVariables > 8 ) // http://www.ndepend.com/Metrics.aspx#NbVariables
NUnit Code Maintenance and Evolution
Finally let’s also mention that the NUnit team is serious about maintenance and evolution: more than half of the code base has been refactored between v2.4.1 and v2.4.8. To read this metric view know that:
- Each rectangle represents a method
- The surface of a rectangle is proportional to the number of Lines of Code of the method
- Rectangles in blue are those matched by the CQL query below. In this context rectangles in blue are methods that have been refactored or added between
v2.4.1 and v2.4.8.
