Darrell Norton's Blog [MVP]

Sponsors

The Lounge

Wicked Cool Jobs

Syndication

Recent Posts

Tags

View more

News

  • Darrell Norton pic

    MVP logo

    View Darrell Norton's profile on LinkedIn

    Currently Reading:

    weewar.com

.NET User Group

Articles

Archives

Advertisement

Darrell Norton's Blog [MVP] » Please stop the fragmentation of the testing tools market!
Images in this post missing? We recently lost them in a site migration. We're working to restore these as you read this. Should you need an image in an emergency, please contact us at imagehelp@codebetter.com
Please stop the fragmentation of the testing tools market!

Jamie Cansdale, brilliant VS.NET addin programmer, posts about a System.Testing unifying framework (one testing framework to rule them all!):

Peter Provost has started a petition to include unit testing support in all versions of Visual Studio 2005. While I agree that unit testing should be essential part of all development, I don't think Visual Studio is the place to introduce an important new API. I think a much more fitting place would be the BCL and SSCLI/Rotor. At the moment the BCL has an extensive set of unit tests. I would be in favor of converting these over to using a core testing framework (say 'System.Testing'). We would then have a standard API for writing tests under all implementations of the CLI. I think this is where real value would come.

Having said that, there is going to be a need for unit testing support in standard versions of Visual Studio 2005. At the moment I am developing a testing tool what will work in all versions of Visual Studio and integrate with Visual Studio Team System. It will support NUnit, csUnit, MbUnit and the testing framework that comes with VSTS. The idea is to allow those using VSTS to run their existing unit tests unchanged via the Team System extensibility hooks. It will also allow those not using Team System to run tests written using the Microsoft testing framework.

So long as Team System remains a premium feature of Visual Studio, there is a point carrying on with this and other projects. I am relieved that Microsoft have chosen to tread relatively lightly on the thriving .NET tools community. I think this can only help encourage innovation and cross fertilization.

Why are people creating all these new testing frameworks? Four different frameworks is at least two too many. While there might be something interesting in csUnit or MbUnit, they aren’t compelling enough for me to leave NUnit (in marketing terms these are known as switching costs). And since I’ll be using Visual Studio Team Server, I’ll very likely be using that testing framework due to all the integration work Microsoft has put into it. Why can’t we roll any functionality from the non-market leaders into the market leader (which I would assume to be NUnit, but pick one and everyone support it)? Is it pride that gets in the way? Really, the important part is to make it easier to develop better software.

UPDATE: This is not to say that Jamie is fragmenting the testing tools market. I just don't think there should be effort spent on supporting three different open-source/freeware testing tools. One tool, like Jamie proposes, is a great idea!

Posted Mon, Jun 14 2004 7:20 AM by Darrell Norton

[Advertisement]

Comments

Steve wrote RE: Please stop the fragmentation of the testing tools market!
on Mon, Jun 14 2004 3:49 AM
I agree, if there is something valuable in csUnit, of mbUnit they should be integrated into NUnit. Whether or not they do have something valuable I don't know becuase like you NUnit is the only tool I'm using for my unit testing.
Darrell wrote re: Please stop the fragmentation of the testing tools market!
on Mon, Jun 14 2004 4:05 AM
Steve - thanks for the support! We're all very busy, so let's collaborate to make our lives easier! :)
Jamie Cansdale wrote re: Please stop the fragmentation of the testing tools market!
on Mon, Jun 14 2004 4:37 AM
I would say 4 different frameworks is 3 too many! What we need is *one* minimalist framework with extensability points to allow for more advanced unit testing (say you need to run specific tests in a new app domain).

If a unit testing framework is introduced to all levels of Visual Studio, we will be doomed to at least 3 testing frameworks. One used in the SSCLI/Rotor, one used by Visual Studio (Microsoft.VisualStudio.QualityTools.UnitTestFramework) and one used by Mono and everyone else who doesn't use Visual Studio (probably NUnit)!

Testing frameworks aren't exactly rocket science. It would be possible to agree on a standard through the proper channels (i.e. the BCL/SSCLI) that everyone would use. Let's not mess it up by making 'Microsoft.VisualStudio.QualityTools.UnitTestFramework' a de facto standard!
Darrell wrote re: Please stop the fragmentation of the testing tools market!
on Mon, Jun 14 2004 5:10 AM
Jamie - I agree with pushing the support down to the CLR. That's what Microsoft did with Generics to make sure they worked acorss all .NET implementations!
Jim Arnold wrote re: Please stop the fragmentation of the testing tools market!
on Mon, Jun 14 2004 5:36 AM
It's really not that easy. There are different frameworks because different people have different requirements. There does not appear to be a one-size-fits-all testing tool (for instance, some people want setup/teardown methods at the fixture level, some at the individual test level; some want to specifiy ordering of tests, some think that's bad idea; some want a new instance of each fixture for each test, others don't; etc.). I would challenge anyone to come up with a "standard" testing API which doesn't spark huge debate within the community.

I would also challenge anyone to convince Microsoft to just ship NUnit (or whatever) instead of (yet again) reinventing the wheel.

Jim
Darrell wrote re: Please stop the fragmentation of the testing tools market!
on Mon, Jun 14 2004 5:52 AM
Jim - I think all of that can be supported in a single API (the first two items you mention are already supported by NUnit). We don't need different programs out there that only support a subset of testing requirements. That's like saying we should have different .NET frameworks due to different preferences in how we deal with the file system.

Second, as for MS just shipping NUnit, they won't because they integrate it with all their projects. The level of integration that will come out-of-the-box is much greater than any addin currently on the market. I'll bet that the testing tools they come up with will be much better than NUnit. Not to knock on the open source tools, but Microsoft can commit enough people that the end results are better.
Jim Arnold wrote re: Please stop the fragmentation of the testing tools market!
on Tue, Jun 15 2004 2:39 AM
"We don't need different programs out there that only support a subset of testing requirements."

I think that's naive. Sometimes it's not what you put into an API but what you leave out that counts. Some things have not made it into NUnit because the project team (and some of the more vociferous users) feel those things would be open to abuse, or go against the philosophy of the framework. What would you do in that case - have certain implementations throw NotImplementedException for the features they feel are dangerous? Just because it can be done, doesn't mean it should be.

"Second, as for MS just shipping NUnit, they won't because they integrate it with all their projects. The level of integration that will come out-of-the-box is much greater than any addin currently on the market"

Maybe that's an argument for Microsoft to open Visual Studio up for greater integration with 3rd party tools.

"Not to knock on the open source tools, but Microsoft can commit enough people that the end results are better"

Throwing more people at a problem does not guarantee good results. For example, I happen to think Visual Studio is pretty poor in a number of areas, given the number of people they throw at it.

Microsoft are very new to the idea of test driven development, and the thought of them stomping all over that space riding their integration elephant (however helpful they think they're being) makes me rather uneasy :-)

Jim

Darrell wrote re: Please stop the fragmentation of the testing tools market!
on Tue, Jun 15 2004 2:55 AM
Jim - it looks like we are at the "let's agree to disagree" stage.

I still think everything can be rolled into 1 (or 2) programs, be it NUnit, csUnit, mbUnit, or whatever. Not implementing something because the developers think it will be abused has failed in the past. :)
Shawn A. Van Ness's weblog wrote re: Unit Testing without Dependencies
on Thu, Oct 21 2004 6:04 AM
Devlicio.us