Jay’s TDD QuickStart, and the underlying problems he stumbled into

Jay Kimble, CodeBetter's resident AJAX guru, issued a little challenge to us TDD bloggers about using Test Driven Development to develop a custom extension to the MS Ajax ScriptManager control.  In the comments, Jeff Perrin (you need to blog more Jeff) proposes a solution that's about what I would have suggested — remove most of the functionality to a "POO" class that isn't coupled in any way to the ASP.Net runtime.

I think the most important things to know or think about upfront before transitioning to TDD are:

  • There is a way to write an automated test for almost any piece of code, including the user interface itself (the idea that the GUI can't be tested is somewhat a myth)
  • The biggest part of driving code through the tests is thinking through *how* you're going to split up the responsibilities of the code to be able to easily write unit tests *before* you commit to writing code.  I know a lot of people blanch at the thought of doing things just for testing, but a lot of making code testable is simply a strong focus on separating responsibilities into cohesive and loosely coupled modules.  Basically, traditionally good design in general.
  • Assume that you will have to change the way you code in order to make writing the unit tests easier.  Maybe you won't, but you should still have an open mind and be prepared to challenge the way that you write code.
  • Specific to .Net, the "Visual Studio way" of creating code often detracts from testability.  More on this below.


Laying down the Law

Back to Jay's issue, let's try to apply some of Jeremy's Laws of TDD.

  1. Isolate the Ugly Stuff.  In Jay's case the ugly stuff is anything related to the ASP.Net runtime.  Following law #1, boil the "meat" of the real functionality away from the "bones" of the ASP.Net runtime.  Now, the functionality can be tested without all the overhead of a web server. 
  2. Push, Not Pull.  In Jay's control, divide the responsibilities into two pieces.  The control itself is only responsible for handling the interaction with the ASP.Net runtime and delegating to the "POO" class that actually implements the functionality.  If the "POO" class is given everything it needs to do the work, it's completely decoupled from its data source, and therefore easy to test.  Known input, expected outcome.  This is what I call the "Roadies and Aerosmith" analogy.  The "POO" class (Aerosmith) is doing all the sexy work, but the web control (the Roadie) is responsible for giving everything to the "POO" class that it needs to do its work.
  3. Test Small Before Testing Big.  Obviously, the whole thing, control and "POO" class, needs to be tested front to back in an integration test.  Before you even attempt the big bang test, build and test the constituent pieces in isolation first.  That strategy both leads to effective evolutionary design, and keeps the debugger turned off. 
  4. Avoid a Long Tail (forthcoming, but soon because I need some content for my DevTeach talks).  Jay's scenario is really too simple for this law to come into play, but think on this:  if you need to pick up any piece of logic or behavior in your system and write a test for it, how much "stuff" has to come with it?  For example, if you can't test some business logic without a database or Active Directory authentication, you're TDD efforts are going to be much more difficult than they could be.


More Classes?  Yes!

Again referring to Jay's post, one of the commenter's is concerned that the extra classes will create performance/scalability problems and I'm would also guess an unvoiced concern over extra complexity.  Two points:

  1. I vehemently oppose the attitude that more classes automatically equates to complexity.  I think a much, much more pervasive problem in the wild is classes that are bloated in size with unrelated responsibilities.  In a way, I'm personally more concerned with the complexity of any one single piece of code more than the whole.  I'll take a dozen classes with cohesive responsibilities versus a single 1000 line procedure any day of the week.
  2. I will happily stand up and shout this all damn day.  In all but the most extreme cases, the performance overhead of creating well factored code (smaller classes and methods) is trivial in the grand scheme of things.  Maintainability, of which testability is a crucial component, should never be sacrificed at the alter of performance without very good reason.  I cringe every time I read somebody say to avoid creating more objects as a performance optimization.  Sure it is, but compared to a single inefficient database query or chatty communication?


The Underlying Problems of TDD in the .Net World

Now, the very real problem that Jay's little example exposes to broad daylight:  in .Net development, and especially ASP.Net WebForms development, you often have to go out of your way to create testable code.  To steal a phrase from Jeffrey Palermo, .Net does not help you "fall into success" with good separation of concerns and testability.  In Jay's case, you can't simply write a small unit test for the control as a whole because any UserControl is tightly coupled to the ASP.Net runtime.  You can fake the HttpContext and the page event cycle, but it's nontrivial and doesn't save you much time (and no, don't ask me exactly how.  The examples I've seen relied on lots of Win32 calls and quite a bit of time with Reflector.  My advice is to go to MonoRail or wrestle WebForms into an MVP structure).  You *can* abstract away the HttpContext by surrounding it with abstracted interface wrappers, and I do recommend this, but the better question I should probably be asking, even as a fervent TDD advocate, is "why do I have to do all this extra stuff for testability?"

One of the most popular posts I've ever written was on using a Model View Presenter architecture with ASP.Net for testability.  Half a year later I completely lost my enthusiasm for MVP with WebForms in a short morning of presentations from Prag Dave Thomas on Ruby on Rails.  What I saw in only a couple hours was a far more elegant solution for testable presentation code than MVP/WebForms can possibly provide.  At one point I even made Dave stop and repeat a sentence on how state and information was moved from controller to view because their technique eliminated most of the mechanical work we were doing to force ASPX pages into something testable. 

One of the huge advantages of Ruby on Rails over WebForms is the inherent testability that is baked into Rails.  The way you develop pages in Rails out of the box is inherently testable, in WebForms you have to apply an extra layer of stuff that isn't built in.  Rails comes with built in extensions into the Ruby xUnit tool to stub out the equivalent of the HttpContext at will to simulate user requests, plus more extensions for common assertions for Controller to View interaction.  You can even happily test the generated web page itself without starting up a web server, and that's a huge win in terms of the rapid feedback cycles you want with TDD. 

Rails straightjackets you into an MVC breakdown by default that separates responsibilities for easier testing.  ASP.Net is all too happy to let you throw everything into the code behinds.  I was extremely dubious about Software Factories at first, but there might be some room for a software factory for ASP.Net that leads to better practices.  We'll see.

If Jay had been building this functionality with Ruby on Rails (which already has that functionality btw), he wouldn't have had to go out of his way one single bit because Rails is inherently testable, and that's my whole point in this section.  We want ASP.Net to have the same level of testability that Rails does.  We should be asking Microsoft to move ASP.Net in a direction that enables us to write testable code without fighting the framework.

I don't know how much say or influence I get with this MVP thingie, but the one single thing that I would beg Microsoft for in whatever is beyond Orcas is far more consideration for testability in the .Net framework.  It's the one area in particular where I think the .Net framework needs a lot of improvement.  ScottGu, since you're amazingly all seeing, are you out there?  Can we talk about testability in ASP.Net?  Can we jump testability over new wiz bangy RAD capabilities in the priority list?  Who's with me?

I really think the .Net community needs to reexamine and debate the merits and appropriateness of the Visual RAD approach.  A lot of what makes ASP.Net WebForms hard to test is the extra complexity that's there to support design time controls, specifically the page event cycle.  RAD is doing more damage than good because the negative impact on maintainability/testability overwhelms the "write once" advantages of RAD.  Discuss.  


My Stuff on Test Driven Development

About Jeremy Miller

Jeremy is the Chief Software Architect at Dovetail Software, the coolest ISV in Austin. Jeremy began his IT career writing "Shadow IT" applications to automate his engineering documentation, then wandered into software development because it looked like more fun. Jeremy is the author of the open source StructureMap tool for Dependency Injection with .Net, StoryTeller for supercharged acceptance testing in .Net, and one of the principal developers behind FubuMVC. Jeremy's thoughts on all things software can be found at The Shade Tree Developer at http://codebetter.com/jeremymiller.
This entry was posted in Maintainability, Test Driven Development. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • amita sexana

    Access File Repair
    access password recovery
    Access Password Recovery Software
    MS Access Password Recovery
    Access Password Recovery Tool
    Fix Excel
    download free key logger
    Chat Archive Recovery
    Database conversion software
    MS excel repair software
    Data Recovery Tool
    Web Hosting
    Free Data Wiper
    Digital camera photo recovery software
    Disk Recovery Software
    Data recovery software free download
    Database Converters
    Free Keylogger
    Excel File Recovery
    MS Access Password Recovery Tool
    Excel Recovery
    free keylogger software
    Floppy Recovery
    Repair Excel
    Excel Repair
    recover msn password
    Excel File Fix
    Floppy Disk Recovery
    Windows Data Recovery
    Keystroke Logger
    Hard drive data recovery
    Free Excel Recovery
    Windows Recovery Tools
    IE Password Recovery
    IPod Recovery
    Key logger
    Excel Recovery Tool
    download a free keylogger
    keyloggers free
    keylogger freeware
    Keylogger Spy Software
    keylogger software free
    keystroke capture
    Recovery Password
    free keylogger downloads
    best keylogger free
    msn password recovery
    Outlook Express Password Recovery Software
    password finder
    Keylogger Software Download
    Password Recovery
    password recovery software
    password recovery software free
    pen drive data recovery software
    Pen Drive Recovery
    PowerPoint Repair Tool
    Chat Recovery Software
    Fix Excel File
    Restore Excel File
    Email Recovery
    cell phone sms recovery software
    recovery for excel
    Repair Excel Files
    repair excel files software
    Data Recovery Software
    sim card deleted sms recovery
    SIM Card Data Recovery
    Key logger software
    downloadable keylogger
    download keylogger freeware
    free invisible keylogger
    data recovery
    Download Free Keylogger
    keystroke recorder
    software keylogger
    remote key logger
    sim recovery
    Sim Card Recovery
    Backlinks Checker Tool
    remote keylogger free
    Spy Keylogger
    USB drive data recovery
    Free Backlink Finder Tool
    Zip Repair Tool
    Data recovery software for NTFS
    Recovery Format Data
    Partition recovery software
    Backlinks Checker
    Sim Card SMS Recovery
    Word file repair software
    data recovery services

    Thanks for this usefull information, i really need it.

  • http://twitter.com/jeremydmiller jeremydmiller


    You can’t, or not easily enough to make it worth the effort. You’re probably better off moving to some other kind of code structure that facilitates unit testing. There’s a reason why so many of us got off WebForms as soon as we could;)

  • Jaisal Surana

    Hi Jeremy,

    Could you please tell me how to do TDD using Nunit or NUnit Asp in code behind pages of a web UI application with session and application variables?

  • Richard F

    It seems as if your tagcloud/ site is broken at present. Keen to read your CI posts. Sort it out mate!! 😉

    – Richard

  • http://www.incisif.net Frederic Torres

    Man, what a post and nice exchanges.

    At InCisif Software we tried to make asp.net app testing a little bit easier.

    Give it a try.

    InCisif.net is an automation tool designed to implement client-side functional testing of web applications under Internet Explorer 6.x or 7.x, using the C#, VB.NET or IronPython languages with Visual Studio 2003, 2005 or express editions.

  • http://blog.tieredsolutions.com Darrell Russell


    I see where your coming from, if I’m developing an uncomplicated solution all by myself then, yes, I can throw all the testing out of the window .. code to the spec (if I have one) and figure it all out in my head. The smarter you are the more complex the solution can be and this “technique” will still work.

    However, there is a point at which this kind of attitude simply dies a death. If the system involves, say, a team working on different parts of a solution that is even mildly complex then sooner or later if your not following some kind of test strategy it’s all going to end in tears and frustration.

    I’m guessing the work your talking about is the former rather than the latter?

  • Andy Alm

    I realize I’m jumping into this discussion a little late, but I have to say that post gets me real excited. Jeremy, the points you make in this post are right on IMHO, and this discussion is something I think is long overdue in the ASP.NET world.

    I think your best point was asking the question, why do I have to go to so much work to test ASP.NET? Sure, its possible as ScottGu pointed out above with an excellent link, but why is it so much work, and where is the guidence on how and why to do this stuff? I have yet to find an ASP.NET book on the shelf that goes into these issues with any kind of depth. And in fact, most of the code samples you see in books and MSDN do all kinds of work in the code behind. Rarely is there a mention of separation of concerns with an MVC or MVP. Contrast that with Rails…its all baked in. I have yet to see a Rails book not talk about testability. And the whole framework pretty much forces you to separate out your concerns into a Model, View, and Controller. Sure, its still possible to write crappy Rails code, but it nudges you in the right way, while ASP.NET does the exact opposite.

  • Carlton

    “Code free development”?

    And I was concerned about job security just a short while ago?

  • Carlton

    OT – listening to some of the comments here I am reminded of Weinberg’s “Secrets of Consulting”: if you cannot say “no” to your clients, then they likely have too much of your business. Not being able to say “no” often times leads to not giving them the information they need to hear.

  • Dee


    If I bid jobs based upon full blown development with TDD and documentation, I never win the bid – period.

    I’m not saying debugging is free. I’m simply saying writing TDD code is too expensive for the ASP.net world. There are simpler less expensive ways to get the job done, at least in my experience. I don’t know about the desktop development world because I don’t do that.

    In websites, that I know, they are constantly growing in scope, changing in features, and updating technology. This to me makes them short-lived.

    Just consider the short life of the ASP/IIS/sql server technology versions.

    MS is moving towards code free development and that seems to be inconsistent with the concept of TDD, which further questions the value of doing this work.

  • http://weblogs.asp.net/astopford Andy Stopford

    Scott to answer your question. I use MVC to test actions on the controller and model at a unit level first then test at a functional level next. UI interactions between the UI (lets say JS into the DOM) are tested at the functional level but anything that involves the UI and the controller has been tested in isolation first.

  • http://weblogs.asp.net/astopford Andy Stopford

    Scott – The trouble with testing large but you test small is that you have no idea what impacts what. If code a calls code b but code b has a defect that code effect code a, and you would have no idea whats going on (and out comes your debugger\tracelines to find out). Testing small means you test in isolation so you can be sure your code behaves as it should, a unit at a time, time after time. I’d like to see the metrics for Rails\MonoRail projects that bake in unit level testing to see how they compare to defects found at a fuctional level.

    Dee – your counter arguements both here and my blog are around cost so lets explore that. It costs for you to find bugs, it costs for you to fix bugs, it costs for you to add new features/change things and by the very nature of doing so prehaps causing bugs – repeat. Websites are not that short lived unless its a very simple site, for lets say a ecommerce site its a non stop process of development where a defect could cause a business a lot. The process of QA is aimed at saving the business money both in downtime and in development time.

  • Dee


    Unfortunately, I still have to pay my bills and eat. The answer is quite easy, I don’t get paid to say “no” .

    There’s not that much work out there that I can pick and choose my customers.

  • http://exceptionz.wordpress.com Maruis Marais

    Dee, Don’t you think it is time for us as developers of these websites to start saying no to the site owners? I’m currently in a contract developing an ASP.NET site and we have a manager who thinks that he can always reduce the estimated time down by 30 – 50%. I’m sure he also believe that he can put nine woman in a room and produce a baby in a month.

    I’ve also found that less-experienced developers tend to say ‘yes’ more often without realizing the knock-on effect their decision will have on the overall health of the project or source code.

    What I think we should do is explain why we need to practice TDD, CI and OO principles when developing their web sites. At the end of the day, these practices will save them money and allow them to have a much superior web site. I’ll quote my existing contract again, at the moment they have so much technical debt in their code base that the new functionality added had such a low coding velocity it was scary. As an example, we had to add two additional fields to the User Registration section, this trivial task took 3 days!! Why? The existing code had multiple duplications of the exact same functionality. The in-house development team believe that creating an abstraction layer for the functionality would increase complexity.

    What do you think?

  • Dee

    The webforms /pages I develop are tested simply by adding trace code to the page to confirm information being processed.

    It’s simply on or off.

    Since most web site owners won’t pay for documentation, let alone TDD, this has been the simple effective solution.

  • http://codebetter.com/blogs/jeremy.miller jmiller


    I’m mostly doing WinForms development at the moment (w/MVP and testing views through NUnitForms). On my last ASP.Net project, we used a strict Model View Presenter pattern architecture, tested the controllers in isolation, then and only then hit the UI itself with Selenium RC & FitNesse.

    I’d argue with you somewhat on the effectiveness of MVP and controller testing. There’s a lot of code in any nontrivial web app that has nothing to do with the final html generated, and its vastly better to test *that* code without going through th e UI. Testing small before testing big is the most cost effective way to test an application in my book. That’s why I think it would be such a big win to do much more isolated view testing without having to fire up IIS and run a whole web page.

    Granted, I still think the MVP technique is much more effective with WinForms, but all the same I still see problems getting removed from the code faster with MVP than without. The reason I don’t think MVP is as effective in WebForms as it could be is that the WebForms “abstraction” leaks. Mocking the view isn’t really enough, because you always end up with some debugging time getting the event cycle just right.

    I’m at the MVP Summit next week if you’d like to discuss this more.

  • http://weblogs.asp.net/scottgu ScottGu

    Dee/Jeremy/Andrew – how are you testing your views today? Specifically, are you testing the output of your HTML and the JavaScript interactions in the browser?

    As I mentioned in most origional comment above, if you aren’t testing your view, then you really aren’t testing your UI.

    Testing the controller/code-behind alone, while ceretainly better than nothing, is a relatively small part of testing UI, and all quantitative data I’ve seen indicates that it identifies only a small number of the total bugs in a web UI.

    I’m not tryin to make a MVC vs. MVP vs WebForms argument. What I am arguing is that regardless of which approach you pick, you need to look at the real-world quality implications of your testing approach.

  • http://codebetter.com/blogs/jeremy.miller jmiller


    Obviously I’m going to disagree with you;)

    Couple things Dee,

    1.) I think developing with TDD is faster than developing without, otherwise I certainly wouldn’t be in favor of using TDD. Yes, it is easier to do TDD with WinForms and server code, but I think you can work more efficiently in the whole with TDD doing WebForms than WebForms without TDD.

    2.) Actually, I think you supported my arguments a bit. WebForms isn’t all that testable without force fitting an MVP structure in. Testability, and the relative simplicity in comparison to ASP.Net, is what would attract me to MonoRail.

    “There’s nothing efficient about trying to do TDD development with webcontrols”

    Let’s make it be efficient, that was kind of my point.

    There’s another post in here…

  • Dee

    This article/discussion seems to have lost it’s perspective.

    Websites rarely have the luxury or the budget to develop in a TDD driven concept. The lifespan of a website is too short to support the cost. TDD development is taking the approach of desktop development and is flawed in it’s belief that owners will accept the cost of TDD driven ASP development over the cost of opensource MVC development.

    There is no MVC model in ASP.net. Scott’s complex solution described above only exacerbates the cost issue and reflects the perspective of a developer who works for a company like Microsoft or Google with unlimited cash on hand.

    Real companies have to cut costs everywhere and can only afford to develop efficiently. There’s nothing efficient about trying to do TDD development with webcontrols.

    It’s a square peg in a round hole problem.

  • http://www.codebetter.com/blogs/jay.kimble jkimble

    I just wanted to publicly thank Jeremy for responding… Ok, I baited you Jeremy… I knew you couldn’t resist, and I’m glad you did respond…

    I read this post this morning and now I think I need to re-read it… dang! Lot’s of good info here.


  • http://weblogs.asp.net/astopford Andrew Stopford

    Apologies for the typos, no way to edit the post so instead…

    Doing IT testing in the webform world is just about the only way you can test a webform so its very commonly done. It does NOT make it right though, and for all the reasons above its something that would be better to change.

  • http://weblogs.asp.net/astopford Andy Stopford

    Another point :)

    >>What I think would be interesting to measure in an application is what percentage of bugs you find/fix using UI integration testing vs. mock-based HTTP tests

    Doing that form of testing is only one scale of testing, I would be doing this testing AFTER unit testing and not before it and not to replace it. To test a webform view/codebehind I have to test it using integration testing.

    people have found that the integration UI testing approach has a significantly higher bug found per test ratio, and also seems to yield a higher rate of return when identifying regressions.

    I want to test at the unit level to help me test one piece at a time, integration testing means I am testing several different things at a time. Can I ensure that my code coverage is enough with a IT and will all IT give me the coverage I need. IT also means your testing several different things with different context’s etc is your testing really effective enough at the IT level to cover all the possible combinations of context’s? Really you want to test one peice at a time because it’s simpler to do and you should’nt ever need to worry about context at all. You can also ensure that your tests for the unit of code covers enough to ensure its tested to its fullest and is done so with needing to worry about everything else around it. When I am happy with the unit level then I can move to integration testing and ensuring that it meets the needs of my application and nothing beyond it.

    Doing IT testing in the webform world is just about the only way you can test a webform so its very commonly done. It does make it right though, and for all the reasons above its something that would be better to change.

    Breaking away from Agile for a second, the importance of patterns and testing on the unit and integration level is a corner stone of things like the V-Model and RUP. Certainly in the V-Model it’s a requirement that you unit testing comes before IT so thats not something you would miss.

    Returning to patterns, a pattern like MVC\MVP seperates the concerns and creates the abstraction you need to test effectivly. Just as Jeremy says Rails bakes in MVC so you have all of these things. No one seems to have mentioned Castle yet so let me say that you get the same thing in MonoRail and its ASP.NET.

  • http://weblogs.asp.net/astopford Andy Stopford

    This is something I have said several times of my own and others blogs and its worth saying one more time in the context of this post. Fundmentaly webforms leaks abstractions. Breaking off the layers to give clearer seperation of concerns is one approach but still the view and controller are bound to each other. I can’t TDD operations into my controller easily because its bound to my view so tightly. Creating a extended controller/presenter is one approach but its messy, really messy when the abstraction should be in place in the first place.

    The pipline approach to context is one approach but surely it makes more sense to unseal context and provide a interface type to mock against? I want my context class to be a plain old object and not need to jump through masses of hoops to mock it.

    I am not sure if Mix is the right venu, PDC prehaps? but I’d like to see an open discussion forum that can show case some of the solutions to using patterns and improving testability in the ASP.NET world.

    It’s great to see Scott in the discussion, I know of no-one else who works as hard as he does to address these kinds of concerns. Lets keep the discussion going…

  • Steve


    I think your comment deserves it’s own post – I recommend posting it on your blog or something.

    There is alot of good information in there.

    Brings to mind: there is soooo much information out there, it’s hard to get it in one good spot, I do think this is all good for the .net community – the more we learn about the different ways people are addressing the same issues, the better we call become as developers.


  • http://www.ayende.com/Blog Ayende Rahien

    If you look at Jeremy’s list of required features, we can do it all today (Castle.MonoRail.TestSupport), by basically hosting the ASP.Net runtime in process.
    The problem with this is that:
    – You have no way to provide alternative config file to ASP.Net, “web.config” is literally hard coded into the runtime.
    – Performance is extremely prohibitive, I kid you not when I say that I find testing via the browser to be on par with this approach.

    My own list include:
    – Can’t create a usable instance of a page outside of ASP.Net context
    – Can’t carry out useful conversation with the page (several interactions with it)

  • http://weblogs.asp.net/scottgu ScottGu

    I think there are a couple of aspects to think about testing UI in general, and especially web UI that is split across both a client and server tier.

    One important aspect is to structure your code in such a way that you separate your UI logic from your business/data logic. When I hear people say that they have 100+ lines of code in either a code-behind class or a pure MVC controller, I always cringe.

    Ideally you should keep a code-behind file down to event handlers that on average don’t have more than 4-8 lines of code in them. These lines should be “glue code” that do the work to pull/push values in/out of UI controls and pass them to methods implemented in classes that are UI agnostic. There should be absolutely no non-UI/non-interaction logic in them.

    The separate UI agnostic classes are where your business logic and data logic should live. I recommend always putting these in separate class library projects. By being UI and host agnostic you can unit test them using any standard class-level unit test framework.

    For testing the UI portion of your application, you have a couple of different options. One approach is to go the Mock route and simulate code-paths in your server-side UI logic. There is definitely a lot of benefit to this (there are also a lot of minuses – more on this later). The two big benefits I see are that you can usually run tests very quickly, and that this approach is probably the easiest way to get high code-coverage numbers of your server-side code.

    It hasn’t historically been well documented how to simulate HttpContext objects for testing ASP.NET requests (although interestingly the hosting APIs have been there from the beginning to enable it). Dmitry and I put together some sample on how to-do this last fall, and shared it with Steve Smith and a few others who have put together a CodePlex project that is adding to the concept. You can check it out here: http://www.codeplex.com/plasma

    The above code allows you to create an ASP.NET application in-memory (without requiring a web-server or a separate process), and allows you to simulate http requests to ASP.NET (effectively mocking HttpContext objects through the pipeline). This will actually execute the full ASP.NET pipeline (allowing you to simulate authentication/authorization, \bin path code loading, configuration, dynamic compilation, caching, session/profile management, and ultimately HttpHandler/WebForm behavior). The guys on the project above are working on a set of client-control test utility classes that allow you to easily simulate postbacks and UI control interactions against the server as well.

    You can then use them within a standard NUnit or MBUnit project to test your pages/applications – no real HTTP traffic required (you don’t even need a web-server installed). The positive side is that you can also process hundreds/thousands of requests a second – which makes running lots of tests quickly very easily.

    Because you are simulating HTTPContext Mocks, it also allows you to write your tests first (before you even have a page on the server) and do true TDD.

    As I said above, this approach has a lot of pluses, and I do think for fine-grain code-coverage it is a good approach to employ within your projects.

    The one (big) caution I’d have is that simulating code-paths on the server-side alone isn’t always/usually the best way to test that the application really works. It does a great job of marking your lines of server code green in a code coverage tool, but by doing so it can lull you into a very false sense of quality in your application.

    Especially in the world of AJAX, but even in a non-JavaScript environment, you need to worry *a lot* about the client-side browser side of your applications. Too often people miss this in their excitement of getting high server side code coverage numbers.

    For example, you might have 100% code-coverage on the server, run all your unit tests and see green and feel great, but accidentally have a mistyped html element close-tag within your .aspx/.htm/.php or Rails .rhtml/.erb view file and have your application completely fail for every single request to it. Despite having 100% server side code-coverage and all your unit tests passing, the browser will trip over the close tag (for example on a form element) and not parse/handle input correctly.

    As another example: Almost every web application has client-side JavaScript today (whether for focus management, client-side validation, or richer AJAX scenarios). You can again have 100% code-coverage on the server, but if your client-side JavaScript has an error with the name of an HTML element, or a mis-typed line of obscure JavaScript, you are going to not have the page work correctly (despite all your mock unit tests passing on the server-side piece).

    Instead of using only a mock approach on the server, I’d recommend always having at least some end to end UI integration tests that you incorporate into your project. You can use a replay tool (like what is in VSTS) for this, although increasingly I’m seeing .NET developers leverage programmatic unit testing frameworks like WatiN that provide more control and flexibility.

    If you haven’t tried WatiN, I’d recommend visiting these two links to download it and try it out:


    You can incorporate WatiN within NUnit, MBUnit or the VS test framework and use your standard unit testing run tools with it (it allows you to use C# or VB to write the tests as standard NUnit or MBUnit test cases). It does launch and use the browser to test the application, so it won’t be as fast as zipping through 5000 class-based unit tests. But it will give you exactly the experience that you’ll have when a customer hits the site, which ends up being a far more accurate assessment of the application quality. It also allows you to fully test AJAX and JavaScript scenarios.

    What I think would be interesting to measure in an application is what percentage of bugs you find/fix using UI integration testing vs. mock-based HTTP tests. I’ve only seen data from a few projects so far (and would love to hear more experiences), but from what I’ve seen measured, people have found that the integration UI testing approach has a significantly higher bug found per test ratio, and also seems to yield a higher rate of return when identifying regressions.

    I wouldn’t argue that this replaces the need for Mock based UI server code testing (which again has the benefit of being able to be run much faster than automated browser testing, and so can be invaluable with quick refactoring regression checking). But I think it does point to:

    1) the need to have both types of tests within your projects

    2) the need to quantifiably track and measure the ROI you are seeing within your test suites, and potentially adjust the ratio of tests you write based on it.

    3) the need to make sure teams and team members internalize the importance of making sure the application actually works, as opposed to being purely focused on code-coverage numbers.

    For Mock based Http testing of ASP.NET (both WebForms and non-WebForms requests), definitely check out the Plasma project above. I think it has the potential to give you what you want to doing functional unit testing on the server of ASP.NET WebForms and UI controls.

    For integration testing of ASP.NET (or JSP/PHP/Rails) check out either WatiN (which provides a .NET object model) or Watir (the origional which uses a Ruby model) for doing more end to end UI testing within a browser.

    Both Plasma and WatiN can be integrated into any .NET unit testing framework today.

    Hope this helps,


    P.S. Note that in VS Orcas the Unit Testing framework and test running features are being moved down from the VSTS SKU into the Visual Studio Professional SKU. This will make them much more broadly available, and allow them to be integrated into any Visual Studio project.

  • http://www.hdri.net Chad Myers

    You said: “I know a lot of people blanch at the thought of doing things just for testing”

    I used to feel this way, but it seems more and more that the tests serve a second, ulterior purpose: They are, essentially, the first consumer/client to your API. If you’re having problems writing unit tests, you’re essentially experiencing first-hand what consumers of your API (including poor maintenance programmers in the future) will have to deal with. If it’s painful for you, and you were the guy who wrote it, imagine how much more painful it’ll be for the guy who has to figure it out from scratch.

  • Steve


    And I think you are right – I really like what RoR brings.

    I’m surprised Ayende didn’t bring up Monorail – it’s an excellent way to go! I just wish it could get the backing it deserves!

    Great set of ideas there as well.

    I’ll be 100% honest: I tend to avoid testing the UI at this point – not because I don’t want too, but more because it’s so difficult to do – both in asp.net and winforms. Anything to help in these regards would be beneficial!

  • http://codebetter.com/blogs/jeremy.miller jmiller


    I certainly agree with you in regards to layering the ASP.Net user interface to maximize unit testing, but there’s a bit farther we can go to make unit testing easier and extend the reach of unit tests before we have to go to WATIR or Selenium.

    I’ve used Model View Presenter designs with ASP.Net to promote testability, but I don’t think it’s the ultimate solution. MVP is meant for stateful UI’s, and try hard as it may, ASP.Net isn’t stateful.

    What I’ve seen from RoR is a high degree of testability without nearly as much mechanical work for testability as we have to do with MVP and mock objects in ASP.Net. I’m thinking that a simplified eventing pipeline ala MonoRail or RoR (get input, do something, build model, view template turns the model into html) with a templating engine that can run and be verified in process is the way to go. I’ll grant you that I might be asking for too much given that WebForms is already established and probably locked into that eventing pipeline.

    Specifically, what I’d like the ASP.Net team to consider is:

    * An easy way to simulate HTTP requests
    * Have a quick way to stub the HttpContext in memory
    * Be able to run user controls and web forms in some sort of lightweight container where you can easily control the incoming state and check the output. Basically, some easy way to run/stub the event pipeline somewhat completely in process. At least
    enough that you could test the HTML output of Jay’s control without difficulty.

    * This could be OSS, but a set of assertions on top of the lightweight container that could let you declaratively express test conditions, just like RoR’s test extensions.

    I apologize for the RAD crack, that was a bit out of line.

  • http://intrepidnoodle.com Aaron Robson

    I’ve blogged about similar things before –
    http://intrepidnoodle.com/blog/show/1.aspx – and the more I’ve done to overcome the issues with webforms, the more I’ve come to understand and appreciate some of the issues which aspnet is trying to solve.

    Without going into it too deeply, it seems that a lot of the architectural decisions stem from the attempt to make user interface reuse possible via controls. If nested forms were an option, I think a lot of things would have been nicer. Other things that stem from this are PostBack and ViewState – neither of which are inherently bad, and solve specific problems. (But why create a LinkButton which needs javascript and use it for navigating via postback ???).

    The bad stuff seems to begin when they went beyond the perfectly useable Repeater control and HtmlControls. It’s at this point, with things like DataGrids where an awful lot of behavioural code starts getting embedded within their user interface controls, and they start becoming tied to the IDE and visual designers and overly complicated Databinding. Perhaps they could start shipping those controls split into separate assemblies for the none web related part of them. On second thoughts, I don’t care, I don’t really intend to use them anyway!

    As ScottGu points out in the article I mentioned above, asp.net has many levels for us to use. I just wish they’d stop pushing people to use that top level of absolute trash which is built purely for the IDE, all it does is lead to lots of unmaintainable declarative code. ‘Oh great, another article on using the GridView on Msdn’ – just what I wanted :) When the evangelist speaks – ‘Just say no’!

    Wow, I could rant for hours on this :) I’ll stop now before I get onto aspnet Themes and start spitting at passers-by.

  • Steve

    By the way, Shawn Burke posted using Atlas test harnesses they created:

  • Steve

    I was playing devil’s advocate, because you are asking Scott Guthrie to provide you with the ability to test asp.net

    1. I can be meticulous to separate my UI from my other layers. The other layers can be tested with NUnit – nothing Scott needs to provide

    2. I’m left with how to test the UI. That comes down to how I design the UI. ie. you mention MVC, etc… based on that design will help determine how I test it. At this point, I’d use Watir or something similiar.

    What are you asking for ScottG to provide?

    I feel we do work on similiar projects, I just haven’t ask Scott for a way to test my code yet :)

  • http://codebetter.com/blogs/jeremy.miller jmiller


    We’ve had mixed results. I’ve had some success in testing from the Presenter/Controller level down to the database with a mocked view in FitNesse on past projects, and I’d happily recommend that.

    On my current project we used a lot of NUnitForms, both in isolation and inside FitNesse. I’d say the results were mixed. We found and fixed some bugs, but we’ve since found that it’s far more efficient to only unit tests controls in isolation. MVP does make that testing much easier.

    Even just testing the controller classes and isolating every thing you possibly can from the view machinery helps quite a bit. I went after ASP.Net, but the data binding in WinForms seriously klooges up automated testing through WinForms UI’s.

    One thing I’d like to play with someday (after I finish StoryTeller & my DevTeach talks), is to try to use the RubyCLR tool to create an equivalent of WATIR for testing WinForms. Failing that, I’m an admin on NUnitForms, and we could supercharge that a bit.

  • http://codebetter.com/blogs/jeremy.miller jmiller

    Steve 2,

    You guys passed the “Law of Steve” – in any given group of developers there will be more Steve’s than women.

    “What if I want RAD and I don’t want tests?”

    A.) You’ve already got RAD
    B.) You just can’t work with me, but before you get too offended, I bet we work on very different types of projects

    “Does every developer, everywhere, everytime, need to have tests?”

    – The bigger problem is that I do want to use TDD and have a very high degree of unit test coverage, but ASP.Net and .Net in general have some room to improve in terms of enabling that style.

  • http://codebetter.com/blogs/jeremy.miller jmiller


    You’re pretty well right on the money in terms of mocks. The best way to use mock objects is probably worth a short book someday, but all the advice I have to give is in the “Mock Object” links above. The biggest point of using a mock object is to create boundary conditions to allow yourself to write smaller tests that run quickly. There are other tricks (Inversion of Control) to isolate functionality for smaller unit tests that can be better alternatives to mock objects. To isolate business logic from the database, I think a Domain Model approach + a Database Mapper approach gives you the best ability to unit test business logic independently of the database. That way you don’t even have to use mocks in most cases.

    As far as a variance between the mocked behavior and the real thing, you shouldn’t try to mock an interface that you don’t understand. No matter what, you will need a level of integration tests as well. Even so, I would contend that it’s faster and easier to remove the majority of problems in code through the smaller unit tests than it is through larger integration tests (limiting the variables in any one test). I haven’t had many problems with drift between mock and the real thing, but that’s dependent on staying within the rules – don’t mock interfaces you don’t understand, don’t mock fine grained interfaces, etc.

    On the database itself, add a series of tests that test just the DAL. I like to do at least smoke tests for sanity’s sake with an O/R mapper as well. Before you go into integration tests you should have a good idea that every single piece behaves as expected.

    The efficiency in terms of developer time is from the unit testing reducing debugging and bug fixing time more than the time spent to write the tests and maintain them. That equation is favorable when you can make the code easy to write tests for, otherwise it doesn’t compute.

    And if you have a wacky legacy database, all bets are off.

    thanks for the comment,


  • http://www.ayende.com/Blog Ayende Rahien

    > The idea that the GUI can’t be tested is somewhat a myth

    It can be tested, but often the investment just isn’t worth it.
    And I have yet to find a tool that will assert that all my labels and fields are aligned on the same line…

  • Steve

    I don’t think this will go over very well, but you are enforcing that things must be tested.

    Perhaps that is the argument. Does every developer, everywhere, everytime, need to have tests?

    What if I want RAD and I don’t want tests?

  • Steve

    Excellent post.

    Interesting that you mention Rails and how it’s ‘baked’ in.

    Unfortunately for Microsoft, even if they baked it in, you’d be stuck needing to buy their development suite :)

  • http://www.e-Crescendo.com jdn

    As the ‘commenter’ on the other post, I should point out that I didn’t mean ‘scalability’ as a performance concern, but whether or not it is a ‘scalable’ development practice to have to create multiple classes, objects, etc. each time you want to test something like what Jay was describing because of ‘leakage.’

    I should have used a different term. But, to clarify a bit more (hard to do in a blog comment but I’ll try):

    Since I’m more familiar with mock objects: I think I understand them. If you are doing testing, you don’t (theoretically) want your tests to fail if the dev database server is down, or some other environmental condition, and so you mock out what the database would be returning you so you can make sure that your test tests just the piece of functionality you want tested.

    But how do you know the mock objects are in line with the database? I can imagine all sorts of scenarios where there is leakage/drift/call it what you will, where your mocks happily work with your code, but once you hit the database, the code fails.

    And if you are going to go full in with mocks, you’ll probably end up with hundreds of additional classes, hundreds of places where you can get drift. In which case, it might not be so bad to just test against the database directly, since that’s what your code is going to do in production.

    So, do the ‘stand-ins’ lead to problems down the road? That’s my concern. The one test for Jay’s scenario is simple and manageable. Multiply it by 20, or 200, and then I wonder.