Code Contracts and Automated Testing are pretty much the same thing

Automated testing became mainstream something like 5 years ago. Nowadays, in 2010, automated testing is somehow accepted by most serious developers.

Design by Contract exists for 2 decades, first supported by the Bertrand Meyer’s Eiffel language. Thanks to the recent Code Contract .NET v4 library, there is nowadays, some buzz about Designing by Contract in the .NET community.

My point is that one developer shouldn’t abide by one of these practices, without adhering to the other. Especially, if you already practice automated testing, you ethically cannot ignore Code Contracts. I have the chance to be relentlessly serious in practicing Code Contracts AND Automated Testing for 5 years now. My conclusion is that, in appearance different, Code Contracts and Automated Testing represent pretty much the same thing.

Code Contracts and Automated Testing share both the same primary goals:

Check often for code behavior correctness
Trigger obvious failure in case of broken correctness found
Document code behavior

Assertions put inside the code through Code Contract are really the same thing than assertions put inside test code. This advocate for a central point: Code Contracts assertions should raise obvious failure when violated during the execution of some automated tests.

Often I hear or read that just few tests covering a large portion of the code is not a valuable thing, since few tests contain few assertions and thus, there is a high chance for a bug to remain undetected. In other words, some pretend that high test coverage ratio of the code is not so much valuable. But if the large portion of the code contains a well-written set of code contracts assertions, the situation is completely different. During the few tests execution, tons of assertions (mainly Code Contracts assertions) have been checked. In this condition, chances that a bug remains undetected are pretty low. Chances that a bug remains undetected are pretty close to 0 when the code is both 100% covered by tests and contains an optimal set of contract assertions.

More concisely Contracts are not a replacement for Tests and vice-versa: both should be
practiced seriously.:

Contracts need Automated Tests to be exercised often in a repeatable way.
Automated Tests need Contracts as a much finer way to check code behavior correctness, inside the code itself.

Finally, some might discuss that unit tests are class oriented, that integration tests are modules oriented that Design by Contract is class collaboration oriented, bla bla bla. While I agree that there should be a clear frontier between the set of quick unit tests (ran in seconds), and the set of slow integration tests (ran in minutes) I am not so strict about the scope of tests and contracts. My mission as a developer is to write bug-less code in a competitive time. Relentlessly applying both automated tests and Design by Contract is so far, my best weapons to achieve this task.

http://kun.io/ William P. Riley-Land

This is great. I’ve always felt based on experience, that functional tests are wonderfully helpful in finding regressions, but also that unit tests were usually to be avoided as offering little gain. You’ve pointed out that it’s because if you are doing Contract Programming you’re already doing unit tests. Brilliant!
http://www.massqa.com/ Outsourcing Software Testing

I find very un usual and very relevant to other software testing.
Dan

When you talk about 100% coverage, you are talking too about UI code, web service calls, file system operations, database calls… ?
http://codebetter.com/members/Patrick-Smacchia/default.aspx Patrick Smacchia

Melissa, before .NET v4 contracts were coded with
calls to System.Diagnostics.Debug.Assert(…) methods. A limitation is that assertions call could only be invoked in Debug mode, there were not invoked at production time.

Now .NET v4 comes with a powerful Contract API with dedicated tools, see more about it here:
http://msdn.microsoft.com/en-us/devlabs/dd491992.aspx

With this API + tools, it is up to you to decide when the contract assertions code is executed (test-time, debug time, production time…)
http://irrsinn.net Melissa

Hey, Patrick. I love these testing/DbC posts. I’m learning and exploring automated testing in projects I run on my own, and these provide valuable insight.

When you talk about code contracts in cases like this, do you mean literally putting assertions in your code using System.Diagnostics.Contract or PyDBC? Code that would actually execute?

When I’ve read about DbC in textbooks, it’s always been described in a vague, nondescript way that suggested you would use code comments to describe the pre- and post-conditions, rather than executable code.
http://codebetter.com/members/Patrick-Smacchia/default.aspx Patrick Smacchia

Gunnar, sure contracts have a cost in terms of code size and in terms of performance.
Personally, I have contracts activated on Debug build and contracts removed on Release/Production build.
This is not optimal since we loose a lot of potential verifications at production time but a choice has to be made here.
http://weblogs.asp.net/gunnarpeipman/ Gunnar

It seems affordable to use contracts instead of tests but there are some gotchas. If you go for 100% code coverage with automated tests then you write basically a separate project with many lines of code. This code is targeted only to testing your code and that’s it. But this code needs also system resources like memory and CPU time to run. If you take medium or large project and look at all these tests (unit, integration, tracer bullet etc) then I am very sure you don’t want your system to be bigger beast than it is right now. To me it still seems good point that I made once: you have to test also your contracted code because contracts may have bugs (http://weblogs.asp.net/gunnarpeipman/archive/2010/05/06/code-contracts-unit-testing-contracted-code.aspx).

I played with code contracts and thought about how I will use them in live scenarios. I found that I will cover with contracts the situations that are most likely to come true in some situations. But I am still sure I don’t want to cover 100% of my codebase with contracts. If you look at contracts after compilating your code you should think twice how many such monsters you want to see in your code. Example is here and it makes me a little bit nervous: http://weblogs.asp.net/gunnarpeipman/archive/2010/05/22/code-contracts-how-they-look-after-compiling.aspx
http://codebetter.com/members/Patrick-Smacchia/default.aspx Patrick Smacchia

>The other thing to look at is Pex from Microsoft research

100% agree! Pex is cool and very relevant in this topic.

>When you say automatic testing, do you mean automated testing?

yes indeed, I fixed that, I am not native english, thanks

>There are two huge differences:

>1) Verification code in test code doesn’t add complexity to your production code.

Personally I find contracts represent great documentation and relieve complexity. It makes code much easier to read and understand.

>2) Assertions in test code run when the tests are run. Assertions in production code may not run until customer runs your code.

Hence my point on 100% code coverage. 100% code coverage means all contracts exercised often.

At the very least we can expect contracts assertion to be ran during smoke tests, before code is going to product. If it is not the case, if some contracts get never exercised somehow before going to production, then the development shop suffers from a serious testing problem.
fschwiet

When you say automatic testing, do you mean automated testing? As in, unit tests? I assume thats the case but it sounds like you may be talking about something else.

“Assertions put inside the code through Code Contract are really the same thing than assertions put inside test code.”

Thats false. There are two huge differences:
1) Verification code in test code doesn’t add complexity to your production code.
2) Assertions in test code run when the tests are run. Assertions in production code may not run until customer runs your code.

You suggest that your tests are set up to ensure all contracts are hit, addressing #2. I don’t think thats quite sufficient. Tests need to be self-contained.
If a test relies on contracts existing to validate behavior of the system under test, someone can change the contracts in the code without realizing they’ve invalidated a test.
Nick

I think one point is worth making.

Most tests involve setting up the preconditions, then asserting the post conditons and invariants.

It’s redundant to code test assertions in most cases if you have contracts, because you are just repeating the code.

ie. The tests should be exercising the code, and the contracts asserting that things are working as they should, with any tests asserting particular values expected.

The other thing to look at is Pex from Microsoft research

Nick

Code Contracts and Automated Testing are pretty much the same thing

Archives

Friends of CodeBetter.Com