How should your automatic tests behave when they are executing an assertion through  System.Diagnostics.Debug.Assert(…)?

I have been mulling over this question because by default, the tool TestDriven.NET ignored my assertions. Concretely, while executing tests with TD.NET, the Assertion Failed! window doesn’t appear for violated assertions. You are certainly happy that the Assertion Failed! windows appear when you’re doing your manual tests, then why would you disable this behavior in your automatic tests?

I asked my friend Jamie Cansdale (the guy behind TD.NET) how to remedy this behavior and it is as simple as executing the 2 following lines in your test appdomain before executing any tests:

System.Diagnostics.DefaultTraceListener listener =     

   (System.Diagnostics.DefaultTraceListener)

   System.Diagnostics.Trace.Listeners[0];

listener.AssertUiEnabled = true;

Let me now explain why I wished this behavior.

• Why do we use assertions in our code? Because C# and VB.NET don’t provide yet facilities to write contract.
  
• What is a contract then? Simply put, a contract is a condition that should never be violated. If a contract is violated, it means that there is a bug somewhere.
  
• Then, should automatic tests be allowed to break contract? According to me, the answer is no. Automatic tests are here to detect bugs, they should not simulate bugs.

Let’s analyze a concrete example:

public class Foo {

   public static void PublicMethod(string s) {

      if (s == null) { throw new ArgumentNullException();}

      // ...

   }

   internal static void InternalMethod(string s) {

      Debug.Assert(s != null);

      // ...

   }

}

Automatic tests should test that PublicMethod() raises an exception when its input argument s is null. PublicMethod() can be called by tier code that me and my team don’t know about. It is a common defensive code pattern that protects our Foo framework from misuses.

However, automatic tests should not test InternalMethod() with a null s argument because it is an internal method. As the developer of the InternalMethod() I put this assertion because in my business logic there is no sense to call InternalMethod() with a null argument. In other words, if at a point InternalMethod() is called with a null argument, it means that there is a bug somewhere in my company code (because only code from my company is allowed to call non-public code). If an automatic test is able to trigger a call to InternalMethod() with a null argument, it means that there is a bug somewhere in my company code and I certainly want to know about it.

Of course, enabling assertion during automatic tests can lead to broken build process. My opinion is that this is a risk that is worth being taken because an assertion that fails necessarily means that there is somewhere a bug, a flawed contract or a flawed automatic test.

Notice that in our particular Foo example, if non-nullable types were supported in C# (as I strongly advocated for here) there wouldn’t be any question and any null argument bug would be discovered at compile-time.