Lessons learned from the NUnit code base

I recently analyzed NUnit v2.4.8 with NDepend.

The first impression is that developers behind NUnit know their job and did an excellent work. This positive feedback comes from many details, the fact that you can seamlessly open the VisualStudio solution and compile everything immediately, the amount of documentation, the high number of releases, the number of tests, the support for every version of .NET (and also non-MS ones), IDE independent code…

 

In terms of Lines of CodeNUnit is relatively small with 12 559 LoC (not including tests assemblies). To be compared to 70 681 LoC for NDepend v2.11,    42 092 LoC for Paint.NET v3.22,   36 143 LoC for NHibernate v2.0,    29 791 LoC for Castle v2.0.3,    474 388 LoC for Resharper v4.1,  and around 2M LoC for the entire .NET Fx 3.5 (including WPF, WCF…).

 

Test Coverage of NUnit

In terms of code coverage, NCover says 47% but some tests didn’t pass while running with TestDriven.NETereHere . Here is a NDepend snapshot of methods more than 90% covered (rectangles in blue represent methods that are at least 90% covered by tests). Without surprise, UI code is poorly covered.

 

 

Assemblies of NUnit: How to compile them 3 times faster?

 

While looking at assemblies, an immediate remark comes to me: is there really a need for 19 assemblies for these 13 068 Lines of Code?


I wrote once about the benefits of having less but bigger assembliesOne of the reason was the compilation time and indeed, the 13K LoC of NUnit + 4K Loc of test projects takes 32 seconds to compile (total of 29 projects). In comparison the 70K LoC of NDepend are compiled in 11 seconds (total of 11 projects)So how does such a gap can be explained? 2 reasons, first the high number of projects/assemblies and second, the use of the Copy Local project reference option:

 

 

 

The VisualStudio Project Reference + Copy Local true option is evil! If you don’t believe me, go and check by yourself: because of this option the assembly nunit.core.dll is duplicated 21 times while compiling the NUnit solution with VisualStudio! It means that it has to be copied/loaded/parsed 21 times by the C# compiler that apparently doesn’t come with a cache/hashCode optimization for such a case.

 

 

 

So I did the experiment of replacing projects references by assembly references. I also

parametrized VisualStudio to output all assemblies in a single folder (menu Project Properties > Build > Output path). The compilation times went from 32 seconds to 12 seconds!

 

The second reason that explains slow compilation is the high number of project (19 project + 10 test projects). Clearly, if assemblies were merged into: few test assemblies, one GUI assembly, one console assembly and one framework assembly, the compilation duration could be decreased from 12 seconds to something like 3 seconds, a x10 factor compared to the initial 32 seconds. Then a tool such as NDepend could be used to guarantee that the structure of the code remain properly layered. And indeed, the code of NUnit is pretty well layered. This is the result of the initial decision to create plenty of assemblies. This can be shown by looking at NUnit namespaces through a dependency matrix, there are only 2 small cycles:

 

 

 

This is a great news compare to the monolithic code structure of NHibernate for example, where all of the 62 namespaces literally depend on all others 62 namespaces (as described here). 

 

Another thing to notice is that this high number of assemblies forces NUnit consumer projects to reference several assemblies and also to deploy all of them, without really knowing if one is missing or not. It would be nice to just be able to reference nunit.framework.dll and be able to just copy/paste it at whim.

Assemblies of NUnit: Insider view

By contacting Charlie Poole, the lead developer of NUnit, to have its opinion before publishing the current post, I had the surprise that Charlie spontaneously confesses that reducing the number of assemblies is indeed part of their upcoming plans. This also come with some justification for each assembly, here are Charlie’s remarks:

  • Two assemblies, nunit.exe and nunit-console.exe do nothing but call the main of the two runner dlls. This is a feature requested by folks who want to run NUnit out of their own exe.
  • The uikit assembly was made separate so that it could be used in alternative guis.
  • The mock and framework assemblies are both referenced by users. But not many users use our mock framework, so they need to be able to reference the one they want to use instead.
  • NUnit uses remoting, with some parts loaded in the test domain and others only in the primary domain. This impacts the minimum number of assemblies you want to have. Our split is currently between util and core, with core.interfaces pointing the way to a future split.
  • NUnit has multiple runners, some we write, some other people write. So engine that runs tests has to be separate for those users.

NUnit Code Quality

Through the prism of classic metrics, the code quality is pretty neat: only 62 methods
on 3 233
slightly exceed classic thresholds
(the report tells 4 388 methods because it includes tier code methods):

WARN IF Count > 0 IN SELECT METHODS WHERE 
!NameIs "InitializeComponent()" AND
                                           // Metrics' definitions
     (  NbLinesOfCode > 30 OR              // http://www.ndepend.com/Metrics.aspx#NbLinesOfCode
        NbILInstructions > 200 OR          // http://www.ndepend.com/Metrics.aspx#NbILInstructions
        CyclomaticComplexity > 20 OR       // http://www.ndepend.com/Metrics.aspx#CC
        ILCyclomaticComplexity > 50 OR     // http://www.ndepend.com/Metrics.aspx#ILCC
        ILNestingDepth > 4 OR              // http://www.ndepend.com/Metrics.aspx#ILNestingDepth
        NbParameters > 5 OR                // http://www.ndepend.com/Metrics.aspx#NbParameters
        NbVariables > 8  )                 // http://www.ndepend.com/Metrics.aspx#NbVariables
        

NUnit Code Maintenance and Evolution

Finally let’s also mention that the NUnit team is serious about maintenance and evolution: more than half of the code base has been refactored between v2.4.1 and v2.4.8. To read this metric view know that:

  • Each rectangle represents a method
  • The surface of a rectangle is proportional to the number of Lines of Code of the method
  • Rectangles in blue are those matched by the CQL query below. In this context rectangles in blue are methods that have been refactored or added between
    v2.4.1 and v2.4.8.

 

This entry was posted in .NET assemblies, C#, Code, Code Coverage, code organization, CopyLocal syndrome, Partitioning. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://www.4444444444444444409hcvw0wvhm9p8hemc.com Preved

    Awesome collection! http://111111111tv0m0vttqay-7vt-vt0-mqva.com ocpaxps 222222 [url=http://33333333333333333sfgwet.com]333333[/url]You straightened up!

  • http://www.NDepend.com Patrick Smacchia

    >To my knowledge, and as I got confirmed now in my
    >little test, VS does not observe the leveling unless you
    >have project references.

    This is where the option:
    Solution Explorer >
    Right click Solution item >
    Project Dependencies

    comes into play :o)

  • http://brumlemann.blogspot.com Martin Moe

    Yep. Just confirmed in my BuildDependencies Mcikey Mouse solution (made it as a proof of concept for the abovementioned MSBuild bug) that going from project dependencies to assemblies dependencies left me stranded in the middle. Can’t clean it and then can’t build it.

    Am I missing something here Patrick?

    You say “So I did the experiment of replacing projects references by assembly references”. It’s been a dream of mine for a long time to do this since then I can have much more freedom in setting up VS solutions on top of our projects (and you know why ;)).

    But how can that work?!

    To my knowledge, and as I got confirmed now in my little test, VS does not observe the leveling unless you have project references. So you end up with missing references as soon as you start cleaning your solution (unless you’re lucky with the naming and happen to have your levels layed out according to the sort order of the names of the VS Projects).

    Martin Moe

    PS Forgot to mention. Nice article, as always ;) Just this little thing nagging me.

  • http://brumlemann.blogspot.com Martin Moe

    Hi Patrick

    Nice to see that we see eye-to-eye on the evilness of “Copy Local” (I actually have a little comment on it on my blog too).

    I am not so sure that assembly reference replacing project reference is always a good thing though. I don’t think the assemly level dependencies will be observed this way.

    One must also remember that there is a known bug in MSBuild (see my blog for more) when it comes to building the clean target of a VS solution (MSBuild cleans .sln projects in the same order as it builds them and not, which would be correct, in the opposite order). Having assembly direct references definitely will not improve on this VS “feature” I think.

    Martin Moe

  • http://www.rasmuskl.dk Rasmus Kromann-Larsen

    Great post Patrick.

    I really enjoy your case studies and the small hints like copy local that come out of them. I just recently downloaded another trial of NDepend (didn’t get time to play around with my last)…

    I was thinking of doing a test run on my own app (for a resulting review blog post), attempting to levelize namespaces and break cycles, if only the trial had direct/indirect mode in the dependency matrix so I could actually find some cycles. But of course that is one of the incentives to buy the commercial product – I guess.

  • Brian

    Excellent what you pointed out about a large number of assemblies. It’s appalling how many times I’ve seen people digging themselves further down a rabbit hole by feeding into this over abstracted multi-assembly structure when not needed.

    It’s a delicate balance between fighting coupling and over architecture.

  • http://www.NDepend.com Patrick Smacchia

    >OK, but how do you handle this on a build server? I guess you have Debug version compiled first and only then the Release?

    The problems that can happen all comes from not in-sync bins. It is one feature of NDepend reporting not in-sync issues (between src files PDB and assemblies)
    http://www.ndepend.com/Features.aspx#WarningsAndAdvices

    You can also have problems if you are using DEBUG conditional symbols but there should be no situations where some Debug APIs are not compatible with Release build. If so, then it is a flaw in the code.

  • http://igorbrejc.net Igor Brejc

    Patrick, I agree that 95% of the time developers work with Debug compilations (simply by compiling in VS or running tests inside VS in Debug mode).

    >> This implies while working in release mode, you need to have your debug dir in-sync.
    OK, but how do you handle this on a build server? I guess you have Debug version compiled first and only then the Release?

    I like some of the things in your approach, but I have to say I’m little worried about the maintainability implications of using assembly references instead of project ones and building to a single output dir. From my experience with solutions that used assembly references this had some hidden effects which often came back to haunt us.

  • http://www.NDepend.com Patrick Smacchia

    Igor, no that doesn’t mean that. Actually Debug compiles to .\bin\Debug and Release to .\bin\Release

    What it means however is that both Debug and Release compilations rely on assemblies in .\bin\Debug. This implies while working in release mode, you need to have your debug dir in-sync.

    This is the way we are doing in our team and the experience shows that we spend 95% of our time working in Debug mode. Release is just use for smoke tests, and actually I should say, Release after obfuscation. Indeed, there is still a tiny chances that obfuscation introduces a problem (this happens once in 2 years and 30 releases because of a minor bug in Dotfuscator that has been fixed since).

  • http://igorbrejc.net Igor Brejc

    >> So I did the experiment of replacing projects references by assembly references

    Does that mean that both Debug and Release binaries are now built to the same directory?