Almost 7 years ago I uploaded the OSS project NDepend.Helpers.FileDirectoryPath on CodePlex. Today I just uploaded its successor NDepend.Path on Github. NDepend.Path represents a major step for the path project. It can now handle pretty much all paths scenarios asked by users during all these years, including:
Prefixing path with an environment variable (this one could be the most demanded feature!) My number one issue with NDepend still persists to version 5 – you still can’t use environment variables in framework folder paths.
Using custom variables/macros in path as in $(SolutionDir)
Support for UNC paths (Universal Naming Convention)
In numbers NDepend.Path is 2.032 logical lines of C# code 100% covered by 552 unit-test, spawned through 77 types. This is not a tiny project and I hope this work will be useful to others in charge of applications that need to handle real-world complex paths managements.
Here is a code excerpt showing what can be done with the NDepend.Path API and how to use it.
In NDepend v5.2 just released, we’ve added a new Paths Referenced tab in the Project Properties panel, that lets the user manage all paths referenced by a project, with a glimpse on how paths are resolved on the current machine. The implementation is all based on the NDepend.Path OSS project.
Last week I was presenting a session covering some practices and tooling we were using to build NDepend at BuildStuff Lithuania 2013. One of the essential practice we are using is Code Contracts and really I was surprised when I asked who was using Code Contracts in the audience. Not even 5 hands up in like 150 pro developers, that are among developer++ since they attend professional conference.
So I do really hope with this blog post to convince you, dear reader, that you should use Code Contracts in order to drastically reduce the number of bugs that hit production and to reduce the overall maintenance cost of the code base you are working on, by like an order of magnitude.
If a contract is violated (hence if the condition asserted is false) we consider that there must be a bug.
The important thing here is that if a contract is violated, you (the developer) should be notified of it somehow, to be able to be aware of the bug and get information to fix it.
Another important thing to note, often not well understood, is that if a contract is violated, you shouldn’t try to recover your application. In such situation the application state is certainly corrupted and there is nothing to do about it, except being agile and humble, and try to fix the bug as soon as possible for future executions.
A very common contract, is when one asserts that a method parameters is not null. With MS Code Contract syntax it looks like:
With Debug.Assert() it looks like:
Do you notice how a contract is also a form of documentation? Everytime I look at a method without contracts I wonder what values are acceptable. For example with the method below I can certainly imagine that the one that has written this method didn’t expect str to be null, but this reasoning is just a guess and maybe the null value is acceptable and the method has just a bug.
I admit that I am a bit extremist, if a null value is acceptable, I put a comment explicitly stating that null is valid. But since I think that null values should be banished, it is pretty rare.
But I digressed. While contracts in C# are I’d estimate 66% about avoiding NullReferenceException, there are plenty of situations where you want to assert something else about your states:
str.Length > 2
str == ‘.’
i > 10
validate an invariant state inside a loop
What is NOT a contract
Often developers are doing something like that and say it is contracts:
But personally I don’t consider that throwing a common exception is a contract. I have several reasons for that:
It is perfectly legal to throw an exception without asserting that there is a bug.
An exception can be caught, and hence you might not be notified of it.
Moreover, when an exception is caught, the application can try to recover, and it can be pretty dangerous to work with data potentially corrupted by a bug.
In a word, an exception is a behavior of your program, it is not a contract! A contract is not a behavior, it is an artefact around your code like a unit-test for example, but wait…
Unit-Tests vs. Contracts
…a unit-tests is made of assertions right? Basically a unit-test define a state, throw this state to a portion of your code, and then assert something on the result. If a unit-tests assertions is violated you can consider that there is a bug (in the application, or in the unit-test). And certainly if a unit-test assertion is violated, you’d like to be aware about it. You might even want to fail a build for that, right?
For all these reasons, I consider that contracts and unit-tests are pretty much the same thing. Both are assertions about your code states obtained while running the code. Who care about when and how the code is executed (debugging time, unit-tests time, smoke tests time, production…), remember the important thing is to be notified of assertions violations, because it means that there is a bug.
Notice a corollary: When a unit-test exercise a code that contains contracts, if a contract is violated you should be notified. Such situation means that there is a bug, either in the exercised code, either in the test code, but there is a bug somewhere and tests must not violate contract! For example it means that if your method asserts that the first argument is not null, then you shouldn’t write a test passing a null reference for this argument.
In 2013 every educated developers do write unit-tests, but almost none of them write contracts, hence this post! We are very very few claiming that contracts and unit-tests are the same thing. I really do hope the situation will evolve in the future because really the benefits of using both massively are outstanding!
Benefits of Contracts
Remember, when a unit-test fails or when a contract is violated it means that there is a bug! Frankly, the opposite is almost true. After using massively contracts and and unit-tests for years, I can say that the absence of assertions violations, pretty means that you have no bugs. Of course there are still a few minor pesky bugs. But by using massively contracts and unit-tests (with very high coverage), maintenance cost is cut by an order of magnitude, really.
Another point that few realizes, is that unit-tests is all about code covered by tests. You can write 1.000 unit-tests and still cover only 10% of the code to be exercised. The number of unit-tests is meaningless. What matters is the percentage of code covered by tests, and also the number of assertions verified. And covering code full of contracts with your unit-tests, is a unique way to multiply the number of assertions checked. So a benefit of contracts lies also in the fact that it makes much higher the number of assertions checked at unit-tests time. By using contracts you make unit-tests much more effective. And this is cool, because while unit tests takes time to be written, contracts takes no time to be written once you took the habit.
At NDepend we use contracts also to test the UI. We have a single huge integration test, that triggers everything that can be triggered in the NDepens UI (and I hope we can agree it is a sophisticated UI). This integration test takes 30 seconds to run. It checks literally millions of assertions, and still this integration test itself just contain just a few assertions. So in practice, contracts are also an excellent and original way to test automatically UI.
And finally, we saw that contracts are powerful documentation.
I do really hope that these benefits sounds good to you! Yes, code contracts work, and they work really well!
Contract and tooling
Contracts makes the job of static analyzer tool easier. For example below, a simple asserts makes Resharper clever enough to notify you of dead code, or of a luring NullRefException.
This benefits isn’t an anecdote. I believe that part of the next decade big things for developers, will be static analyzer detecting non-trivial bugs in the IDE. As you can see Resharper already does it pretty well. But it still doesn’t try to check for states from the chain of callers methods and the chain of called methods (yes it does it for .NET Fx methods and few other cases which is pretty cool, but it is still limited). Contracts is what will make such tool clever enough why? Because with contracts you drastically reduce the number of states the static analyzer have to check, and reducing numbers in an exponential algorithm makes all the difference between something that works practically and something that works theoretically. And hence the tool can focus and report only real buggy cases.
In the same vein, there is also the great Pex tool. I have huge hopes in it. For now there is just a lite version for VS2012 and VS2013, but I guess much more is coming.
Also, soon the C# and VB.NET compiler will be opened thanks to the Roslyn project. It will be easier than ever to write static analyzer because we’ll get all the syntax analysis for free. In other words Roslyn will let tools vendor focus on semantic analysis and this sounds very promising to me.
As a side note, one especially cool situation where I use contract daily, is when you switch over an enumeration with say 3 values. Instead of having 3 cases, one for each value, I like to have 2 cases and one default case, asserting the third value in the default case: The benefit is that the switch flow is free from any ambiguity. Both the C# compiler and Resharper understands it and if all cases returns, it won’t ask for a return statement after the switch ! Notice that the compiler magic isn’t coming from the contract itself. But the cool benefit of the contract is that in the default block we can be certain of the status argument value.
MS Code Contract vs. Debug.Assert()
So what should you use? What do we use at NDepend after years of practices? Actually we do use both technologies, dealing with their pros and cons. MS Code Contracts have huge advantages over Debug.Assert():
The framework is nicely done, with many well named methods to handle all scenarios (Require(), Ensure(), Invariant(), ValueAtReturn()…).
The framework let’s put contract on interfaces members.
The framework works nicely with documentation tooling, and there are facilities to include contracts assertion inside the doc itself.
Some checks can be made at static analysis time, to chase automatically for pesky bugs like finding case where a null value can be passed to a method argument that has a contract.
But in practice MS Code Contracts has a single one significant drawback that doesn’t make it competitive with Debug.Assert(). In most scenarios: it double and even triple compilation time, because by using MS Code Contracts assemblies get rewritten once compiled by the C# or VB.NET compiler. This rewrite is needed because MS Code Contracts is doing not natural act on code (like executable code on interface declaration, that needs to be weaved inside the impl methods).
Because long compilation time is not acceptable, in practice, what I’d recommend, is to use contracts on the public surface API of your application (like NDepend.API for us, just one small assembly), and Debug.Assert() everywhere else! Apparently the C# team is considering compiling MS Code Contracts directly in C#6 (source Wesner Moise). This would be really cool, and would certainly make me change my mind and forget definitely about the usage of Debug.Assert().
You’ll notice that in the list of advantages I didn’t put the fact that with MS Code Contracts, assertions can be checked during production. Indeed Debug.Assert() assertions are removed from assemblies compiled in Release mode. Personally I consider this like an advantage because:
Keeping thousands of assertions checks in production code, can and actually do, significantly impact performance.
I don’t consider clients like testers. But still clients do experience bugs, crash and exception reportd (typically automatically reported). Our job is to avoid the client to experience any bug and from my experience, a well detailed exception report is as good as a contract report in 80% of situations.
Btw, one detail point I didn’t mention, is that a MS Code Contract assertion violated throw a ContractException (which is not a public type) that can hardly be caught. And if you think about it, it makes sense. Jon Skeet explains it well in his StackOverflow answer.
As you certainly know, a Debug.Assert() assertion violated triggers a scary blocking dialog that lets attach a debugger. It is especially convenient for notifying a bug and for starting investigating when running code compiled in DEBUG mode.
A bit of History
DbC or Design by Contract has been invented by Dr Bertrand Meyer in 1986 when he specified the Eiffel language. So contract is not something new and I regret so much that when Microsoft built .NET at the end of the 90′s, they didn’t considered including contract in the IL and Common Type System. Especially taking account the fact that M.Meyer was working closely with the .NET team at that time.
As we saw, Contracts is closely related to null reference check, which still represent like half of the bugs on earth! We know that the null reference concept was deemed as a billions dollars mistake by Tony Hoare its inventor. Here also I regret that the .NET Common Type System doesn’t come with a way to statically enforce non-null reference. I’ve read somewhere that it was the biggest Anders Helsberg regret (if anybody has an url on that… cannot find it anymore, but I am sure I read it … ok it is here thanks Antonio for the link ). And I remember in 2004 when I was MVP C# trying to convince Anders to enhance the CTS with non-null references. The one and only argument against wasn’t so much about feasibility, but about the fact that it would create a tsunami of breaking changes in the .NET Framework 1.1 and one very cool thing of .NET is that breaking changes are pretty limited!
Pff…. it was a much longer post than what I expected, but if you read this conclusion I really do hope you’ve learn something, and that you are curious about using Code Contracts from now to see how much benefits it can bring.
Certainly like every ISV, we have a sense of what users are missing and since the beginning almost a decade ago, we really did our best to implement features users were suggesting. As Jason Cohen summarized recently: If you don’t listen to what customers are telling you to do, you’ll build a product no one wants to use. This sounds obvious, but periodically, being reminded about obvious things is welcome.
With the concept of User Voice my guess is that we won’t see new ideas of features we’ve never thought or heard about. No, my hope is to be able to finely quantify, to measure the need for each missing features. I am really wondering if we will see a set features emerging? This would mean, to some extend, that we failed at listening to users feedback But this would mean also that by offering a User Voice page we improved our listening skills.
NDepend version 5 has just turned RTM today! Big milestone! Last week I talked of the UI relifting we did, but v5 comes also with several flagship features. One of those new feature is about measuring and visualizing development trends. Basically NDepend can log a variety of application-wide code metrics at analysis time. Then the tool can plot historic metrics values on charts and trends suddenly appears:
We were pretty impatient and curious to apply this trending feature on the NDepend code base itself! But the feature has been developed just a few months ago and the trend chart above shows trends for like the past 7 years. To fetch the past values, we did a short program based on NDepend.API that:
Check out past versions of our code base
Pass unit tests and gather code coverage
Analyze the result with NDepend analysis
Log trend metrics (thanks to types in the namespace NDepend.Trend )
Several remarks can be inferred from this trend chart:
This suggests also that our development shop doesn’t suffer from the diseconomy of scale syndroma described well by Steve McConnell in its legendary book: Software Estimation: Demystifying the Black Art : “People naturally assume that a system that is 10 times as large as another system will require something like 10 times as much effort to build. But the effort for a 1,000,000 LOC system is more than 10 times as large as the effort for a 100,000 LOC system”.
As a side note, I also quote this Steve McConnel idea : “My personal conclusion about using lines of code for software estimation is similar to Winston Churchill’s conclusion about democracy: The LOC measure is a terrible way to measure software size, except that all the other ways to measure size are worse. “
My thoughts on high code coverage ratio are becoming prevalent in the NDepend codebase. With v5 we reached almost 80% of overall code coverage by tests. We can see a large effort to jump from 50% to 80% code coverage within the last months and we will continue this effort. What this chart doesn’t show is that code contract is also massively used since I estimate that automatic test and code contract are two sides of the same practice. Once the product reaches a certain critical size it is not humanly possible to sustain a constant productivity, unless you have a good automatic testing and code quality plan coupled with a solid code structure! My hope is that with everything achieved so far, we will increase the productivity. This will be an interesting trend to measure!
The ratio # Lines of Code / # Lines of Comments seems pretty constant, nothing surprising but it is good to visualize this.
The lines of code that fall in the NotMyCode category didn’t increase that much. This can be explained by the fact that the good old CQL impl is mostly generated code, and since then we don’t use much generated code anymore.
That’s a lot of remark for just 4 trend metrics logged. But there are almost 50 default trend metrics defined by Code Query LINQ queries embracing code size, code coverage by tests, max and average values, third party usages (see them listed in the Trend Metrics group). Some out-of-the-box extra trend metrics related to the number of code rules violated and the number of code rules violations, are also available.
Defining a trend metric can be as simple as:
Or more sophisticated like:
Creating your own trend metric is easy if you know a bit of LINQ. For example to count the number of types implementing IDisposable one can just write:
In this post I’d like to focus on some of these UI enhancements. During the last 2 years, in the Visual Studio and .NET ecosystem we’ve seen emerging new UI standards, inspired from Windows 8 standards. With NDepend v5 we followed this trend and took a chance to make most features easier to be discovered, learnt and used.
First we redesigned the main menu. The motto was simple: one feature – one, and only one – word. Icons have been removed from the main menu. Not only because VS API forbids icons on parent menus, but because this would be information overload: words chosen to identify features should be meaningful enough to any professional developer. If for a feature several words make sense, the shortest word has been chosen (Diff/Change/Compare, Trend/Progress/Evolution…).
Below is the Rule sub-menu. We tried to limit as much as possible to one sub-level of menus. For the Rules sub-menu we by-passed this guideline to let the user discover and access the range of default rules, organized hierarchically. Each menu ends up with an option menu (to directly open the feature related option), and one or several links to the related online documentation.
Branding and Logo
Clearly NDepend deserved a better branding and a nicer logo. We are a team of software engineers and I should have admit earlier that we needed this work to be done by a professional designer. Hopefully it is never too late to do things well and we asked the designers of Atomic Duo to do some work for us. I am pretty happy with their results, here are the new NDepend logos:
It is clear that in the Windows task bar it looks much nicer!
With this new logo and the new VS2013 guidelines, it has been easy to make a cleaner Start Page! Atomic Duo are also working on our web-site re-design. Here is the preview, the site will be updated soon.
Ergonomy and User Experience
All dialogs and menus have been refactored to get a smoother user experience, compliant with new VS standards. For example the assemblies (to analyze) chooser dialog have new icons, new button style and new features.
It is important that the assemblies choosing experience is near to optimal, to let the user spend more time browsing analysis result and discover interesting facts about his code base. With prior versions, user was already able to add assemblies from VS solutions, from folders (recursive or not) and by drag&dropping from Windows Explorer. Now we’ve added the possibility to filter assemblies with one or several positive or negative naming filters. Targeting an exact sub-set of assemblies to analyze, among hundreds or thousands of assemblies, is now an immediate task.
VS style has been applied to all panels. Below is the rule editor panel. Not only we have a gain of a few V-pixels, but the overall impression is cleaner. Presenting the same amount of data with less screen area and by increasing the experience: wow, thanks to the VS designers!
We also implemented the idea of having a temporary panel with a violet header. This is a handy addition of VS 2012, that increases the experience when browsing multiple documents in general, and multiple rules in our case. Also the NDepend skinning is now adapted to VS2012 and VS2013 skinning and it gets updated automatically on VS skinning changing. Extra colored skinnings are also supported (VS 2012 only, but I guess they’ll be supported by VS 2013 soon).
In the Queries and Rules explorer panel, clearly the VS2012/2013 large status icons are especially handy to summarize the rules status.
Here also we’ve added a useful UI facility: user can now choose to list all rules violated (or all rules not compiling, or all rules OK, or any sub-set of rules and queries depending on some others criteria):
The shortcuts dialog that appear when hovering the bottom-right progress circle has been also enhanced with new VS 2012/2013 status icons. Listing all rules or queries according to a certain criterion is a matter of a single click on one of these shortcuts:
All panels menus also support new icons. New sub-menus have been added to have a direct access to some advanced features that used to require several steps, only mastered by experienced users.
These advanced features sub-menus are also available from the main menu. Here are some examples of advanced actions, like exporting the namespace dependencies cycles to the matrix. We expect that with this ergonomy strategy, some users will discover already existing features that they didn’t know about.
New Report Experience
The report design has also been enhanced by the Atomic Duo designers. It is not just a skinning enhancement, but also new features appeared with the presence of a detailed Dashboard that summarizes diff, and some Trend Charts.
Since almost a decade that NDepend exists, we had feedback and advices concerning the UI. We noted them all and hopefully v5 addresses most of them. Nevertheless we keep in mind that ergonomy trends are evolving quickly nowadays and we are committed to keep the product up-to-date, while adding new features.