This is a reply to Stephen Fortes post Impedance Mismatch from a ways back. I would have posted about it sooner but I sadly just saw it today when a co-worker Stefan Moser linked it over to me. I know that this debate has become quite heated through the community and as such will refrain from personal attacks (such as those unfortunately experienced by Julia Lerman) and focus solely on the technical merits of the post.

 

My first problem with ORMs in general is that they force you into a "objects first" box. Design your application and then click a button and magically all the data modeling and data access code will work itself out. This is wrong because it makes you very application centric and a lot of times a database model is going to support far more than your application.

 

Well I wouldn't say that this is a problem with ORMs per se but a problem with some tools. Those who are using Domain Driven Design are certainly not using this methodology, one of the main reasons I like to tell people to use DDD is that they can design their data storage mechanisms in parallel to their domain model seeking an optimal solution to each. In other words we should be embracing the impedance mismatch and doing what is best on both sides. The paragraph then continues with

 

In addition an SOA environment will also conflict with ORM.

 

I do not necessarily agree with this in any way shape or form but am happy to leave it left open to "the many definitions of SOA". I think it can quite easily be done if you follow solid command query separation. Udi Dahan gives a nice discussion of this on his blog.

Later in the article (I am jumping around a bit to keep my own post coherent)

 

One of the biggest hassles I see with LINQ to SQL is the typical many-to-many problem. If I have a table of Ocean Liners, vessels,  and ports, I’ll typically have a relational linking table to connect the vessels and ports via a sailing. (Can you tell I am working with Ocean Freight at the moment?) The last thing I want at the object layer is three tables! (And then another table to look up the Ocean Liner that operates the vessel.) Unfortunately, this is what most tools give me. Actually I don't even want one table, I want to hook object functionality to underlying stored procedures. I really want a port object with a vessel collection that also contains the ocean liner information.

 

The author discusses his experiences with Linq2Sql and then applies it to "what most other tools give me", this is an unfortunate fallacy or a lack of research on available tooling. Linq2Sql is not a real "mapper" nor is what the author referring to "mapping", it is simply an Active Record implementation that is not using self-serving objects. This is what happens when mappers stay too close to the relational structure, they suck in terms of domain language and structure.

If we were however to use a real mapper (let's say the one those notorious mafia guys are using) a quite different scenario would exist; a domain that sounds almost exactly like what is described as being wanted. This paragraph is also key in showing that research has not been done into Domain Driven Design by the author, I would bet that Stephen and Eric could have some really interesting discussions at the Advisory Council as Eric uses this exact problem domain as a naive starting point for examples in about half of his book.

A more serious problem is shown though in the authors propensity towards a relational bias when domain objects are called "tables". Why would anyone have a domain full of "tables"? These are behavioral objects. Unless this misunderstanding of what a domain model is is corrected the rest of what a domain model is or does will never make any sense.

A further lack of understanding of Domain Driven Design is shown with the statement of..

 

ORM is real good for CRUD and real bad at other things.

 

Again I believe the author has become confused between ORM and Active Record for some reason. I would never under any circumstances recommend someone to use Domain Driven Design for a CRUD app as there are easier ways (like using Active Record). DDD is hard and often painful, it is costly up front and should only be used in domains that can justify its up front costs in maintainability.

Although it may be surprising, it is my belief that the author is actually a Domain Driven Design aficionado but has just not yet realized it yet.

 

I prefer to build the application's object model and the data model at about the same time, with a "whiteboarding" approach that outlines the flows of data and functionality across the business process and problem set.

 

It is quite common in an "object first" perspective to be either doing database and code modeling either in small iterations or in parallel where a team of object experts focus on the domain model and the best way to model the data in order to support transactional behaviors while a team of database experts focus on how best to store the data given their own set of requirements. These types of sessions would in fact be prescribed in an agile team and the small "whiteboarding" sessions are absolutely prescribed by Domain Driver Design.

 

Maybe it is the MBA talking but I tend to be "business and customers first" when I design a system. (Those of you that know me know that I have designed some very large and scalable systems in my day.)

 

This is one of the core beliefs of Domain Driven Design, the primary example would be the creation of an Ubiquitous Language in order to ease communications between the "business and customers" and the team.

 

What I am saying (and have been saying for a long time) is that we should accept, no, embrace the impedance mismatch!  While others are saying we should eradicate it, I say embrace it.

 

Again we are back into agreement with Domain Driven Design. I like to look at Domain Driven Design as being an orthogonal architecture, my domain survives through anything that is moved around it as it is the core of my business and where the largest amount of my investment has gone...

We come now to where the author is unfortunately not in line with DDD but perhaps can be moved. The only way that one can reach an orthogonal architecture is to ensure the purity of the domain model. The OLTP RDBMS will eventually leave in popularity, what happens when I want to move to say "the cloud" and just store my aggregate roots as XML, this is a perfectly valid and extremely effective architecture. If I favor too heavily the RDBMS side of the impedance mismatch then this change will not be orthogonal to my domain and will as such be extremely costly. The author may disagree with my reasoning as he points out.

 

ORM tools should evolve to get closer to the database, not further away.

and

Developers who write object oriented and procedural code like C# and Java have trouble learning the set-based mathematics theory that govern the SQL language. Developers are just plain old lazy and don't want to code SQL since it is too "hard." That is why you see bad T-SQL: developers try to solve it their way, not in a set-based way.

and

So ORMs are trying to solve the issue of data access in a way that C# and VB developers can understand: objects, procedural, etc.  That is why they are doomed to fail. The further you abstract the developer from thinking in a set-based way and have them write in a procedural way and have the computer (ORM) convert it to a set-based way, the worse we will be off over time.

 

Well I think I have already discussed the first of these points pretty well, by moving closer to the database we break our hopes of an orthogonal architecture. The second comment albeit sounding like it came from a grand and mighty sql wizard sent down by the gods to lift us heathen from our sinful ways is actually a red herring as is the third when framed properly.

I do know relational algebra (yes I can tell you what an anti-join is) and I challenge anyone to show me notation for an insert. While one could argue it can be involved with say a delete by PK/FK or update by PK it is for all intensive purposes useless in the process of writing to a properly normalized database, these items tend to be procedural regardless. I will admit there are times where it can come in handy but they are by far the minority. The relational algebra is focused on reading data and manipulating sets.

As many who have had long post-conference talks over beer with me know I find any query that is of any amount of complexity close to thinking about the relational algebra to be a report. Reports are not expressed within my domain and may or may not be read from the same data source (I often times use an eventually consistent reporting model specifically for the purpose of running such queries). I take this often to extremes, my repositories in an ideal world have a single read method, FetchAggregateByUniqueId. Anything that is searching in a more complex nature is deemed a report and sits outside of this (usually as a small mapper that returns DTOs that match screen shapes, not domain shapes but provide the appropriate aggregate ids for writes to be possible). My "reports" all make very strong use of SQL and Relational Algebra, my domain has no need to know that it exists as it is essentially a write only model. I could go much more into this but it is another post.

Getting back to the article, the author does however end off with a great quote from Ted Neward:

 

"Developers [should] simply accept that there is no way to efficiently and easily close the loop on the O/R mismatch, and use an O/R-M to solve 80% (or 50% or 95%, or whatever percentage seems appropriate) of the problem and make use of SQL and relational-based access (such as "raw" JDBC or ADO.NET) to carry them past those areas where an O/R-M would create problems."

 

This is great advice ... just remember if you do it to hide it from your domain and to use it sparingly as you may not always have a RDBMS sitting behind you and if you don't these set based operations may be quite difficult to implement.

Actively participating in the development of a static code analyzer such as NDepend is a good way to learn about tricky Common Type System (CTS) details. Every feature have to be tested properly to cop with things like managed and unsafe unmanaged pointer types (ref out and *), open/close generic types, generic parameter types, methods with special name (.ctor() .cctor() get_ set_ add_ remove_...), all sorts of arrays combination, virtual/new/override methods, explicit interface implemented methods, nested types, anonymous namespace... And of course, all this tricky points can be more or less combinated.

During our tests we found 2 bugs related to anonymous namespace. One in VSTS Coverage and one in the Visual Studio Code Model. Hopefully, the other CTS peculiarity seems to be bug free as far as our tests are concerned.

When the lightning strikes 2 times the same place, I suppose that this is not by chance: namespace are not first class citizen in the CTS. In other words, there are no metadata tables related to namespace in the metadata of an assembly. At IL level, namespace are just types names’ prefixes. This lack of support for namespace certainly leads team responsible for these features to forget about testing these cases.

The Team Coverage Anonymous Namespace Bug

This bug occurs when there is a nested type in a type declared in an anonymous namespace. For example testing this piece of code…

…leads to the following Team Coverage XML. We can see that instead of having the class ClassInAnonymousNamespace.NestedClass declared inside the anonymous namespace, we have a class named .NestedClass declared in the ClassInAnonymousNamespace namespace.

The Visual Studio Code Model Anonymous Namespace Bug

This bug occurs with VB.NET, when you have a type declared in an anonymous namespace. For the VB.NET compiler, the anonymous namespace doesn’t exist. Indeed, the VB.NET compiler makes it so that every type declared in an assembly are prefixed by the name of the assembly. I don’t know the reason for such choice, but I hope there is a solid one because it is really not elegant. Update 24 July 2008:  This is a customizable VB.NET project setting: Application > Root Namespace.

Now consider the VisualStudio code model. As shown below, a VB.NET type declared in an anonymous namespace doesn’t reference a namespace object. To see the assembly name namespace, one needs to analyze the delta between the FullName and the Name of the type.

It has been only four days since my self-imposed, month-long incarceration in the prison known as Corporate Calgary, but already the three walls of my cell close in around me. My oppressors continue their assault on my psyche with inhuman tortures such as "Death by 1000 Quick Questions" and "Let's Talk About Weather" and "We Meet Because We Can".

My jailors mock me with bizarre acronyms and epic tales of projects past. Yesterday, I nearly screamed in terror when I caught myself chuckling knowingly at one such anecdote. During a meeting to review progress of a system I didn't know existed, let alone used, no less. I fear I may be going insane.

Two things keep me alive. One is the presentation I give next week on ASP.NET MVC to the Calgary .NET User Group on the 29th and the Edmonton .NET User Group on the 31st. During status meetings, I surreptitiously pull up my VM and review my code and notes, ignoring the sidelong glances at my insolent giggles. These presentations are my one tenuous grip on reality and I shall protect them with, well, not *my* life but probably the life of someone close to me.

The second is the one ray of light in my task list: a throwaway application I pull up when I need an extra dose of lucidity. Just this morning, I discovered an error that made me nearly leap for joy for the mere mental exercise, slight as it was:

private void ConvertFile( string converterName, string sourceFile, string destinationFile )
{
    IConverter converter = FindConverter( converterName );
    converter.ItemProcessed += converter_ItemProcessed;
    int recordsProcessed = converter.processFile( sourceFile, destinationFile);
}

This (much simplified) piece of code contained an error. The first time it was called, it ran fine. Each subsequent call appeared to process the file multiple times. I.e. the second time it ran, the file was processed twice, the third time, thrice, the fourth, ummm...quadrice. And so on and so forth.

Alas, the solution presented itself all too quickly. The file was not actually being processed multiple times. Rather, a brand new ItemProcessed event was being appended to the event handler each time through. So the second time the file was processed, the ItemProcessed event was called twice each time an item was processed. The third time, thrice. Etc, etc.

Unfortunately, the fix was trivial:

private void ConvertFile( string converterName, string sourceFile, string destinationFile )
{
    IConverter converter = FindConverter( converterName );
    converter.ItemProcessed += converter_ItemProcessed;
    int recordsProcessed = converter.processFile( sourceFile, destinationFile);
    converter.ItemProcessed -= converter_ItemProcessed;
}

And now I am left to ponder my decision to work onsite for the next month in this soulless place. Left to wonder why I left my safe haven. To toil between a woman whose radio is always slightly off-station and a man with a cruel tendency to squeeze his squeaky, rubber duck at inopportune moments.

Pray for me, readers. Pray. For hope.

Kyle the Indentured

1. What is it?
2. How introducing LINQ to .NET changed the framework
3. LINQ Providers
4. LINQ to XML
5. LINQ to SQL - how it’s different from EF, tips and tricks, when to use it
   

Hopefully though, if you're a fan of the foundation series, you've already downloaded the learning application which puts the theory to practice using ASP.NET's MVC framework. So, what do I think about ASP.NET MVC? Overall I've been very impressed. I can't think of a good reason for starting a new project using the WebForms model - or MonoRail for that matter (sorry). If you're an ASP.NET developer, it's really a no brainer.

I do have two major issues with it though. First, if you come from almost any other MVC framework (MonoRail, Django, RoR, Akelos, etc...) you might be expecting an actually Model framework - instead you get an empty Model folder. In other words, the MVC framework doesn't add anything to the .NET O/R Mapping / DAL story. From Microsoft's point of view this makes sense, since they feel that they are already offering solid solutions - DataSets, SqlDataSources, LINQ to SQL, Entity Framework. Truth be told, this is fine with me, as it lets me use NHibernate. I just think, given what the other MVC frameworks offer, it's a little dishonest - you'll end up disappointed if you're expecting to be able to do this out of the box:

public class Car : ActiveRecord
{
}
....
Car.FindById(1);

My real problem though is simply that neither C# nor VB.NET lend themselves all that well to view logic. Jeff Atwood actually just blogged the same criticism. Jeff uses RoR to highlight the problem. I don't fully agree. I won't say that RHTML is great, but I will say that it's far better than C# or VB.NET. I think views need a specialize language - I'm sure that anyone who's done some significant work in either RoR or Django would agree. There are solutions available now - NVelocity and Boo (I assume you could use it with the MVC framework?), but I'm just going to trudge along with C# until IronRuby is a viable solution.

Aside from that, everything is pretty solid - routes work great, helper methods are adequate (they're starting to add more and more), and testing is actually doable - I haven't run into any problems, but from what I've read things aren't 100% perfect yet (either way, it's a huge step up from WebForms).

So, to recap. MVC good. WebForms Bad. C# in views less than ideal. Empty Model folder = M. Oh, and download the learning application!.

I put together a quick screencast showing how to leverage T4 Templates in Visual Studio 2008 for Code Generation:

• T4 Templates in Visual Studio for Code Generation Screencast

I show step-by-step how to create a simple T4 Template and then how to generate various Entity, Data Access Object, and Factory Classes using more advanced T4 Templates that one might use to create a simple custom data access layer. I have seen other screencasts that have used SQL Server Management Objects, but in my case I use database schema related functionality in good 'ol ADO.NET. Although I wouldn't use this technique to build a data access layer given all the cool code generators and O/R Mappers, it is a pretty decent way to show the possibilities with T4 Templates.

I picked up a new iPhone 3G the other day and absolutely love it, so I also created a version of the screencast that you can download and play on the iPhone. In fact, several of the more recent Unity and Enterprise Library 4.0 Screencasts now have an iPhone version.

If you are interested in some of the ADO.NET techniques I used in the screencast to get database schema information, check out the show notes with links to various tutorials.

I hope the screencast is useful.

Code UnCoverable by Tests

Even for test-coverage addict (as me), there is some code that simply cannot be covered by tests. An example?

The call to MessageBox.Show() cannot be tested automatically since it is blocking. Of course we could mock calls to MessageBox.Show(), but at the end of the day, there will be at least one call to this method in the code base, one call that cannot be covered by tests.

This example is not isolated. There are a multitude of cases where code reacts to things that cannot be automatized, such as exception that cannot be reproduced…

…but also some environment settings, folder or file browsing by user, showing a modal custom dialog and more.

The Problem

As we explained above, while using mocking can help reduce the surface of code not-coverable by tests, it can’t help to get a code base 100% covered. When exploring test coverage results, with NCoverExplorer, Visual Studio Team System or NDepend, this problem will lead to classes 98% or so covered. It is taking time and it is error prone to figured out each time you are exploring coverage results, that these are false alert. This is clearly anti-agile! What we want is a clean 100% coverage result!

The Solution

NCover and NDepend are tackling this problem the same way: you can exclude some methods from coverage results through a dedicated (and eventually custom) attribute.

NCover knows about this attribute through the /ea command line option as describe here by Jamie Cansdale. Code tagged with this attribute will be excluded from coverage statistics and won’t disturb while exploring test coverage results. For example, if all methods of the class Foo are 100% covered except the method MessageBoxShow(), if the method MessageBoxShow() is tagged with the coverage exclude attribute then the class Foo will be shown as 100% covered. A great side-effect is that developers reviewing the code will know about this coverage exclusion.

With NDepend, the coverage excluding attribute can be provided through the import coverage file dialog as shown below. This will have the same effect as with NCover. For convenience, the assembly NDepend.CQL.dll can be linked from your project. This assembly contains the dedicated attribute: NDepend.CQL.UncoverableByTestAttribute. If you prefer you can provide your own attribute.

As far as I know, Visual Studio team System Coverage doesn’t support yet this feature but it might be considered for future releases, as explained here.

The Bonus

Actually the burden of exploring coverage results to make sure that some classes or namespaces should be 100% covered is not really agile. It is manual, time consuming, error-prone and cannot be capitalized for future iterations.

NDepend can help automate this task with just 2 CQL rules:

// <Name>Types 100% covered by tests</Name>
WARN IF Count > 0 IN SELECT TYPES WHERE 
HasAttribute "NDepend.CQL.FullCoveredAttribute" AND PercentageCoverage < 100

// <Name>Methods 100% covered by tests</Name>
WARN IF Count > 0 IN SELECT METHODS WHERE 
HasAttribute "NDepend.CQL.FullCoveredAttribute" AND PercentageCoverage < 100

Basically, these rules will check automatically that all types and methods tagged with the attribute FullCoveredAttribute are indeed 100% covered. Insert these rules in your build process and you’ll be informed as soon as a lack of test coverage is luring. Here also, as a bonus side-effect developer reviewing or refactoring the code will know instantly which part of the code is supposed to be 100% covered.  For applying this trick to the code base of NDepend since the code coverage feature is available, I can testify that it is a really, really time saving trick.

For convenience, the assembly NDepend.CQL.dll can be linked from your project and it contains the dedicated attribute: NDepend.CQL.FullCoveredAttribute (and also the attribute NDepend.CQL.MoreThan95PercentCoveredAttribute). Of course, CQL rules can be readily tweaked to work with a custom attribute.