Unit Test vs. Debug.Assert()

 

How should your
automatic tests behave when they are executing an assertion through  System.Diagnostics.Debug.Assert(…)?

 

I have been mulling
over this question because by default, the tool TestDriven.NET ignored my
assertions. Concretely, while executing tests with TD.NET, the Assertion
Failed! window
doesn’t appear for violated assertions. You are
certainly happy that the Assertion Failed! windows appear when you’re doing your
manual tests, then why would you disable this behavior in your automatic tests?

 

I asked my friend
Jamie Cansdale (the guy behind TD.NET) how to remedy this behavior and it is as
simple as executing the 2 following lines in your test appdomain before executing
any tests:

 

System.Diagnostics.DefaultTraceListener listener =     

   (System.Diagnostics.DefaultTraceListener)

   System.Diagnostics.Trace.Listeners[0];

listener.AssertUiEnabled
=
true;

 

Let me now explain why
I wished this behavior.

  • Why do we use assertions in our code? Because C# and
    VB.NET don’t provide yet facilities to write contract.
  • What is a contract then?
    Simply put, a contract is a condition that should never be violated. If a contract
    is violated, it means that there is a bug somewhere.
  • Then, should automatic
    tests be allowed to break contract?
    According to me, the answer is no. Automatic
    tests are here to detect bugs, they should not simulate bugs.

Let’s analyze a concrete
example:

 

public class Foo {

   public static void PublicMethod(string s) {

      if (s == null) { throw new ArgumentNullException();}

      // …

   }

   internal static void InternalMethod(string s) {

      Debug.Assert(s != null);

      // …

   }

}

 

Automatic tests should
test that PublicMethod() raises an
exception when its input argument s
is null. PublicMethod() can be
called by tier code that me and my team don’t know about. It is a common
defensive code pattern that protects our Foo
framework from misuses.

 

However, automatic
tests should not test InternalMethod()
with a null s argument because it is
an internal method. As the developer of the InternalMethod() I put this assertion because in my business logic
there is no sense to call InternalMethod()
with a null argument. In other words, if at a point InternalMethod() is called with a null argument, it means that
there is a bug somewhere in my company code (because only code from my company
is allowed to call non-public code). If an automatic test is able to trigger a
call to InternalMethod() with a null
argument, it means that there is a bug somewhere in my company code and I
certainly want to know about it.

 

Of course, enabling
assertion during automatic tests can lead to broken build process. My opinion
is that this is a risk that is worth being taken because an assertion that
fails necessarily means that there is somewhere a bug, a flawed contract or a
flawed automatic test.

 

Notice that in our particular
Foo example, if non-nullable types were supported in C# (as I strongly
advocated for here) there wouldn’t be any
question and any null argument bug would be discovered at compile-time.

 


 

This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://twitter.com/superbecio superbecio

    please remember that any thrown exception, even AssertException from the NUnit framework, could be swallowed by your own tested code.

  • Jay

    Very nice explanation. Thank you.

  • http://www.NDepend.com Patrick Smacchia

    Good idea.

    I also agree that very few peoples use assertion and that’s very sad. It is such a simple and efficient way to detect bug early.

  • http://www.chadmyers.com/Blog cmyers

    Patrick:

    I think the problem is that many people simple don’t use Debug.Assert().

    If it doesn’t already have it, I would think that NUnit should have a TraceListener implementation that fails the test if Debug.Assert() fails.

    Here’s some code from the hip (no guarantees, but you get the gist):

    public class NUnitAssertFailTraceListener : DefaultTraceListener {

    public override void Fail( string message, string detailMessage ) {
    Assert.Fail(message);
    }
    }

    Then, in your TestFixtureSetup or somewhere like that, you’d do:

    Trace.Listeners.Add( new NUnitAssertFailTraceListener() );