Unit testing is out, Vertical Slice Testing is in

We have been doing testing for a long time. Some people are practicing TDD, but I think that’s only 46 people in the world and they all follow my twitter feed.


We all know the common issues people have with writing large test harnesses around systems: it seems to take a long time, existing code is difficult to test, subsequent changes to code makes a lot of existing tests fail or become redundant, and the test harnesses often can become brittle, in no small part because of the abusive use of mocking.

As a recovering TDD addict, I used to tell people, like many others do, that the issue with their TDD was that they didn’t do it right. If TDD is too hard, do more TDD! In other words, if it hurts, just bang that leg with that baseball bat a bit harder, eventually you will not hurt anymore.

I will come straight out with it: this approach to unit, integration and tdd testing has, by far and large, failed. It’s over. It is time to move on. I have, and I now do VErtical Slice Testing.

A law of diminishing returns

Whatever your test coverage, there is a point where more tests don’t equate more quality. Actually, let me be entirely honest here, I do not believe that a good test harness leads to better perceived quality software for your users.

The people you are trying to please with a good test harness are the ones using your system that are not on a high bandwidth medium with you. Usually that would be your team. To reduce the risk of them experiencing issues, and to increase the chance of them liking your software, you write tests so that their experience is positive.

A long time ago, I realized that testing everything is not always feasible for me, and I practice TDD all the time. A quick mental review of the teams I worked with in the last 12 years I’ve been practicing TDD also tells me that this fact is rather general.

Mocking considered evil

I’ve never been a big fan of mocking frameworks, for one simple reason. Simple stubbing relies on assumptions on the functioning of the mocked part of your system that rarely match reality, for no one is going to look at all the documentation of an API and decompile the code when writing a one-line stub.

As you progress in your implementation, you may learn about more scenarios you didn’t cover, and provided you are in that tiny little fraction of a percent of people refactoring your test harness as you progress your implementation, you may feedback some of that knowledge in previous tests you wrote and in your test doubles, while caressing your pet unicorn and enjoying a bit of sunshine on planet Equestria. Or if you’re like the rest of us, you probably won’t.

Increasing returns for user interfaces

Admitting to myself that I could not fix those problems (you cant fix stupid!), I started trying to understand how I could bring some of the TDD practices I enjoyed (fast feedback, repeatable tests, etc) while increasing the perception of quality, leaving any theoretical or coverage notions behind me. Instead of starting from the components I want to put under test, I start with what my users, hum, use.

If I provide users with an API, I will start by the API that I decide to publish, and consider everything else implementation details. I will only document and polish, a.k.a. test, what other people may use.

All the same, if my user interface is not an API, but some sort of UI, there is very little reason to care about all the scenarios a component may face that cannot be triggered from that user interface.

The process of discovering your user interface has an added advantage. The answer to most but what if questions about such a top-down approach usually unveils an additional user interface you didn’t quite know you had (looking at you, JSON SPA AJAX <insert buzz word of the day> Frankenstein “web” apps).

This is already an understood concept, and is usually referred to as acceptance testing.

At warp speed, everything is relative

A common issue arises from using existing acceptance-driven tools. Automating the browser is slow, automating a DB is as well, so is the file system. Each of those may also fail for reasons that have nothing to do with your code.

That would make your tests brittle and slow, which inexorably will lead to longer feedback cycles, harder to run tests, which would get us straight back to my introduction and why traditional approaches to TDD have failed.

Acceptance testing recommends, to avoid such a problem, the creation of an alternative interface allowing you to remove the browser from the equation. This is no longer necessary. With tools such as zombie being available for free, you can run an in-memory browser that behaves like a browser, runs in-memory and is increadibly fast. No more Selenium issues on running your automated test suite, no interaction with your operating system’s UI library, it’s all fast and beautiful. And if your user interface is an API, unit testing frameworks and test runners have provided those advantages for many, many years.

External-facing components are now in-memory, making executing our tests fast and reliable by not triggering external systems.

I now apply the same concept to internal facing components. Instead of mocking out the inner-most elements, such as the file system, the network or a database, of my system on a per-method or per-component basis, I use libraries that give me, out of the box, an in-memory version of their system that is functionally equivalent to the real one.

It means an in-memory file system would implement the same locking primitives as the real systems, the same rules around reading, writing or seeking data, and be as close as the author of the library can make it to the real system.

In other words, a VEST-friendly library turns the unit tests of that library on it’s head. The component accessing external systems is developed alongside it’s test-double variant, and both are built to react, error-out and validate calls in the same way. The test double can be shipped. I don’t write the mocks, they come as part of the package.

There are many advantages to such an approach. The author of the library knows intimately the contract he provided you as part of his API. The act of providing an in-memory version means this knowledge is now expressed explicitly, forcing error conditions (which are very much part of a user interface in an API) to be accounted for.

A VEST-friendly library will usually end up testing explicit contracts in their test harness, so we go one step further. A library author can ship the test harnesses that exercise the contract they expose for all the conditions that are known to potentially exist, and once again, we turn test code into shipping code: if the author has written a test harness, and the author builds two components implementing an interface, the test harness for the explicit public interfaces can be shipped, as it’s probably already written.

I believe this process to be recursive and leading to the pit of success, as any library built using VEST in mind will naturally feed the VEST approach.

Vertical Slice Testing

The VEST approach replaces all external system calls by functionally equivalent stable and controlled ones, both at the outer-most layer (whatever exercises the user interface) and the inner-most one (whatever touches the low-level IO APIs).

By using VEST, I can focus on delivering web pages or components that focus on user-interaction, I can run the whole system on every test, and do it very quickly. I can change a lot of inside internal code without breaking existing testing scenarios. And should I wish to implement components replacing the ones in my libraries, I can do that without writing tests, because the test harness is already there, testing the contract for me.

Note: As I’m abroad and not available that much, don’t expect quick answers in the comments section but I’ll try my best. Subjects not covered but not forgotten: my team as a user, availability of VEST libraries, code samples, library author burden, brown field projects, generic crazy-talk, unicorns not being ponies, “you never worked in a team!”, “in an ideal world…”, etc.

This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Pingback: Review of: Jimmy Bogard – Holistic Testing | Robert Daniel Moore's Blog

  • Pingback: Review of: J.B. Rainsberger – Integrated Tests Are A Scam | Robert Daniel Moore's Blog

  • http://www.facebook.com/the.electro.dream Dreamr OKelly

    Good stuff in principle, however I find that when people mock, they mock what they shouldn’t (inappropriate boundary), when they stub they stub what they shouldn’t (stub only what you own), and devs are constantly testing things they shouldn’t be, frameworks, network interfaces, etc.

    When your code is highly coupled, your tests will be too. As you move away from framework code or highly coupled code, I find that my tests become much more useful and robust. Brittleness is the sign of coupling.

    Also, I disagree with tests being for ‘developers’. If your app doesn’t have tests, it isn’t proven. If a user finds a bug and submits it, if you don’t write a test to prove you fixed their problem, then that bug WILL crop up again, and your user will wonder why the hell they spent the time to tell you in the first place. I think they call that Microsofting…..

  • Daniel Kolman

    It’s not about unit tests vs. acceptance/integration tests (or VEST), they are not mutually exclusive. Each type of test serves different purpose. As you say, acceptance (or VEST) tests are great to prove that the application will actually work from the point of view of the user. On the other hand, unit tests are a great tool to develop code and ensure that it will be readable and maintainable. The two techniques (unit testing and acceptance testing) go well together.

    It was a moment of revelation to me when I first encountered this idea in “Growing Object-Oriented Software Guided by Tests” book. You need both of them – acceptance tests to keep external quality, unit tests to keep internal quality. And of course, you need mocks to write unit tests, but you must not use them in acceptance test.

    I even gave a talk on this topic, you can find great chart from the book I mentioned in my slides here: http://www.slideshare.net/danielkolman1/tdd-outsidein

  • Aleksey Izmailov

    Please take it as a criticism only, not trying to be rude or arrogant ;)…

    It sounds like what you are proposing is an escape attempt from poor, highly coupled design with lots of mutable state. You didn’t mention what language you use but let me guess – it’s Java.

    I think by avoiding mutable state, isolating side effects (monads, etc), writing in functional style, and having a library of small composable functions which is easy to test separately you can benefit a lot from TDD. It’s not easy to write in a functional style, and its harder to keep separation of concerns (concurrency, IO, etc), but if you have all that properly done you can have nice unit tests that don’t break that often because of refactoring. It’s easier to understand 10 small test cases rather than 1 huge test, it’s even harder to cover all edge cases. You say they might not occur, and you should not test for it but when someone changes your API a bit you will not know something is broken. It will be up to this person to go check all tests and make sure they correctly reflect reality. On contrary with small tests you will see immediately which component is broken or abused. Just to test your theory try to make small implementation changes that affect functionality, but don’t break your tests. If you get green light then your tests are not good.

    I’m for in-memory components for testing like h2 running in postgres compatibility mode when you have production postgre database, or other services simulating real environment like you wrote. But functionality there will be never 100% the same, so you have QA, integration tests or both to test the rest.

  • Rajesh Guptha

    I do believe that the production code should be tested at various levels, unit/component/integration/manual tests. Each solving their own purposes. And with in memory dependency equivalents pitching in here, the feedback loop could tremendously decrease!

    Has there been implementations of integration test suites using these in memory techniques in the real world yet?

  • Ward Bell

    FWIW, Seb, we’ve been shipping products (see BreezeJs) with in-memory mode for years and I’ve been recommending test tactics along these lines. Consequently, developers testing their components that use our stuff don’t mock us; they load test data and flip the in-memory switch. Works for me anyway. Always felt simple. Are we 100%? No … a gaping hole or two. But pretty decent for the popular scenarios. Not tooting my horn (well, only a little) so much as adding an experienced voice to your proposal.

  • Ben

    “any library built using VEST in mind will naturally feed the VEST approach”

    This is a great principle, and it could make a huge difference to the testability of the whole stack.

    When writing a library, I should ship a functionally identical in-memory equivalent, so that others can use it VESTfully.

    But this needs some momentum, even a manifesto. Until library developers feel under pressure to implement in-memory equivalents, the extra effort won’t be worth their while. So until then I’m stuck writing full-stack automated tests (slow and brittle), a barrage of unit tests (to cover the majority of cases where the full stack tests are too slow) and maybe a few integration tests in between. And after that, I’ll probably be up against a deadilne without time to write an in-memory equivalent of my own library…. how to break the cycle?

    • http://codebetter.com/sebastienlambla/ Sebastien Lambla

      It only takes a few libraries to solve a lot of issues. There’s interesting challenges to make this stuff possible on a large scale (which i’ll blog later in the week about), and I don’t expect most projects to suddenly move to this style. As the comments indicates, some people already do that, and my goal here is to promote this approach by giving it a recognizable name. To break a circle, it only takes one point. :)

  • Nicholas Tesla

    Excuse my ignorance but I have heard the term VEST for the first time by your article. I tried to follow the flow of this article but it did not really give me much hint about what is VEST in first place or why it is useful and finally how it differs from traditional, widely accepted testing methodologies (particularly acceptance testing). Probably I should first understand basics of it and how it can be implemented in small scale then I may have chance to compare it.
    When we are writing about a new methodology, it is very important to have a structured flow in the article otherwise it is very likely that it will not be interested for the reader or the terminologies it contains will be grasped as another set of buzzwords.

    • http://codebetter.com/sebastienlambla/ Sebastien Lambla


      You are correct that understanding an article certainly requires a structure that is easy to follow. I attempted to explain what issues i had, how i solved them and what resulting components I get in the end by applying such methodology (if one can call it that), I’m sorry that it isn’t clear in the text. Any pointers you can provide to where the text doesn’t make clear the why, how and results would be very welcome, shall I ever decide to submit my blog for publication as an article.

      I coin the term to cover a specific approach to constructing a system together with its test harness. As I mentioned, I’ll ship some code to explain more visually. You can apply VEST be it that you formulate your test harness as scenarios, examples and expectations, or that you formulate it per-api being exposed if so is your choice. it does not preclude or embrace being test-driven, even if my practice itself is very much test-driven.

      As for quoting the various testing methodologies around and how my approach differs or builds upon them, maybe I’ll clarify it later on. I make an assumption that the discussion on the specificities of borrowing from acceptance testing, full-stack testing, bdd and the interaction with test-first and tests at some point, is better contained in lists where A/agile practicioners may enjoy an abstract approach to debating more than the audience of a site focused on “coding better”.

      VEST is as much a buzz word as TDD, AJAX, CRM, CQFD (if you’re francophonically inclined) are. It stands for a moniker that describes a set of practices that produce specific results, and describes perfectly the approach I’ve had of building software for a long time. It’s shorter than “acceptance-driven, top-down, with fully-functional test-doubles provided as part external components, and with test harnesses shipping as implementation references”. I’m sure you’ll agree that it’s a mouthful.

  • Dave Hillier

    I’ve found this technique to be useful. It can allow you to be far more economical with the amount of code you need to write and stop yourself over-engineering.
    However, there are problems – it can lead to a combinatorial explosion in number of tests.
    Combining this technique with unit tests can help with this; isolating some parts can help reduce this explosion.

    • http://codebetter.com/sebastienlambla/ Sebastien Lambla

      Dave, I do agree with you in certain instances. In practice, I found that testing edge conditions in lower-level components does tend to result in “composite interfaces”, where for example the same error message is expressed in two command results, often without taking into account the context under which they occur. I find that I end-up rewriting a lot of tests around those boundaries of allowable ranges to provide an interface that “sounds” right, and that makes unit testing those conditions less critical. Although some cross-cutting concerns, as well as using TDD as a design tool for certain components where the results are not necessarily kept after the development of the componenet is finished, can complement the approach nicely.

      • Dave Hillier

        While I haven’t fully formulated my ideas about this, something about the phrase, “cross-cutting concerns”, feels close to it for me.
        By doing TDD at a class level you should be isolating the different concerns (if you can do it right, which I’ve seen plenty of times that there is no guarantee). There are advantages to standardizing like this, but when you standardize you have to accept some compromises – it is never going to be the most efficient approach.
        Sometimes you can encapsulate a vertical slice or cross cutting concern/horizontal slice in a single class sometimes not.
        Vertical, horizontal, class, integration; there is no silver bullet. Each approach has its merits.

        • http://codebetter.com/sebastienlambla/ Sebastien Lambla

          I agree. I tend to classify cross-cutting concerns (someone mentioned security) as another interface of value (after all, XSS is just a user interface where the user is an evil bot). It’s an interesting concept however: if the concern is cross cutting, maybe the test harness ought to be as well. :)

  • jpwkeeper

    While I’m still not clear on how this differs from automated acceptance testing using something like SpecFlow, I’m curious as to how you re-create the file system in memory for VEST. That’s been the holy grail keeping us from using vertical testing in our .NET applications.

    • http://codebetter.com/sebastienlambla/ Sebastien Lambla

      OpenFileSystem. Not quite VEST compliant yet, but the next version may be.

  • Lee

    If one finds a full vertical slice that integrates many components to be LESS brittle and LESS expensive than an isolated unit test, there is a large problem with the way the units are being tested. The most important thing about units is to test behavior, not implementation. I have found that the number of mocks and stubs can be drastically reduced by creating isolated components that are designed to be composed by higher level objects rather than depending on each other directly (this is a functional programming concept). You end up with a lot of simple value in, value out tests. The higher level components then become your target for integration testing. I have found full stack testing in just about any tool to be extremely painful, time consuming, error prone, and fragile and I continually find myself asking if their benefit outweighs their cost.

    • http://codebetter.com/sebastienlambla/ Sebastien Lambla

      What full stack testing have you tried?

      • Lee

        I’ve done cucumber and capybara in larger enterprise apps, and some headless using phantom js and recently angular’s e2e tests on smaller side projects. I have also swapped in memory components as well for speed (sqlite, vcr for recording network traffic). In every case I found that writing acceptance tests that are thorough enough to test all of your system’s complex edge cases is way too costly and cannot possibly replace the value I get out of unit tests that exercise the entirety of the domain complexity. I can not imagine being able to reason about code or refactor without a suite of unit tests to back it up.

        • http://codebetter.com/sebastienlambla/ Sebastien Lambla

          How widespread would you say “unit tests that excercise the entirety of the domain complexity” is actually implemented as such? How many OSS projects do you see that have a test suite that let you “reason about code”? I am making a statement that such approach has so far failed to chatch on much, and continuing telling the wide, wide majority of people out there that they’re doign it wrong is not getting us any closer. I provide here a different focus to start getting better return on the testing investment. It is not to say that I don’t still do TDD or unit testing myself, it is to say that VEST provides as much or more value. Note that anything involving components that fail to make the transition smooth (selenium, in-mem sql lite and so many others) and that prevent you from achieving the goals i’ve expressed here would not be good candidates for a VeST approach.

          To be cheeky: I cannot imagine reasoning about code only through their covering unit tests. Worse, I cannot imagine needing to understand much about implementation details for anythign that is not self-descriptive, well crafted and understandable once the initial investment in learning a system. From experience, while the latter is important, the former is much more important and i would rather invest my limited time and effort allowance in providing better user interfaces and self-descriptive code for what is internal.

          The point is, there are far and few systems that can use VeST due to the small amount of libraries that are compatible with such an approach. The test-first methodology is orthogonal to it. Tests as descriptions of “components” and the definition of whaqt constitutes a “unit” are not covered by this entry. I’m saying that while I consider some of the points you make to be importnat in those two projects a decade I work on that have the qualities you describe, I’d rather spend my efforts adn teachings on approaches that dent into the 99 percent of other projects where unit tests haven’t caught on in the way you describe.

          Maybe the functional world has inherently better approaches. Or maybe the rest of the world is still 10 years away from starting to use functional languages. I don’t know.

          • Lee

            My problem is that saying one technique is “in” and another is “out” is problematic. Its just as problematic as saying that writing SPA’s using frameworks like Backbone are “in” and server side is “out”. At the end of the day what kind of app you are writing should dictate the tools and testing techniques that you use. Most of applications I have worked contain many subtle edge cases in the calculations and operations being performed while having rather simple user interfaces. Trying to exercise lots of tiny edge case paths through the system by driving a user interface has proved foolish in these kinds of applications. In these apps I would be happy with nearly all of my tests being unit tests, and only resorting to full stack tests in more complex user interactions. That said I really could care less what the rest of the world is doing, or what is “catching on”. Using the right tool for the job will stand the test of time.

          • http://codebetter.com/sebastienlambla/ Sebastien Lambla

            Hence the “attention-seeking-headline” comment, right at the top of the article. I don’t sell a silver bullet, I expose what I find to be a more productive use of my time and of the teams I interact with as part of my travels. YMMV. I don’t suggest that this approach is teh silver bullet, but that it reflects what some of the people I’ve interacted with over the years are also applying, and teams I care about think. If you get on fine with traditional definitions of TDD and unit testing, wonderful, and I would be the last one to say that you ought to change. For those other scenarios/teams/environments/times where it doesn’t work, I’m suggesting a formalization of my own coding practice.

            After 12 years of TDD i found something better that gives me more benefits for my scenarios, and I talk about it on my blog. And I shall chose my titles to reflect what I think, for it is my blog.

          • Lee

            Agreed, I wasn’t trying to troll, your opinions are valid. I have just not had such pleasant experiences with full stack testing. Cheers.

          • http://blog.thecodewhisperer.com J. B. Rainsberger

            Your “something better” sets up a false dichotomy with TDD, though. What you do sounds like really good, practical TDD, and it reflects many of the design lessons that I teach through TDD, most specifically thin slices of the product and highly context-independent code. We handle the test double/production implementation/contract problem differently, but that’s merely a question of style, I think. The fact that you emphasise an explicit contract with external resources makes me especially happy.

            Even so, your first significant paragraph gives the game away: “We all know the common issues people have with writing large test harnesses around systems: it seems to take a long time, existing code is difficult to test, subsequent changes to code makes a lot of existing tests fail or become redundant, and the test harnesses often can become brittle, in no small part because of the abusive use of mocking.” I’ve seen too many of these articles, so I have to sigh.

            “large test harnesses around systems” — a problem whether you do TDD or not

            “…abusive of mocking.” — strawman, strawman, strawman

            Now believe me, I strongly relate to “…In other words, if it hurts, just bang that leg with that baseball bat a bit harder, eventually you will not hurt anymore.” That’s just stupid. That’s also not TDD. I know you hate that, but I can’t help it. I hate the “no True Scotsman” fallacy, too, so let me say it differently: anyone who practises TDD, but doesn’t significantly change the way they design to make it easier to write small, focused tests, misses 99% of the value of TDD. They should stop.

            Mocks are making this test complicated? Apply ISP. Apply DIP. Don’t blame the mocks.

            Tests are brittle because people are copy/pasting complicated mocks everywhere? I think copy/paste causes the problem, not the mocks, not TDD.

            I understand that if so many people are doing it wrong, then maybe it can’t be done right. I happen to disagree, but I understand the sentiment. This only shows that TDD isn’t a magic blue pill. That’s no surprise, is it? Neither is VEST, nor any other single technique. I thought Fred Brooks settled that decades ago.

            I think you could recast this article as “After 12 years of practising TDD, here’s how I do it. I use these microtechniques over here and I choose not to use those common microtechniques over there. Here’s why.” You could do this without losing any of the value of this article. (Of course, if your secret plan is to get the anti-TDD audience on your side, then what you did makes perfect sense, even if I find it a bit cynical.)

          • http://codebetter.com/sebastienlambla/ Sebastien Lambla

            There seems to be a common misconception from this entry that I’m putting VEST versus TDD. I don’t. The title specifically talks about “unit testing”. I specifically attack the use or abuse of unit tests, the boundaries currently understood to be units, and the cost curve introduced by that definition of “units” as the software evolves. I heartfully agree about the facts, malpractices and benefits of TDD done right. I’m not, at any point in the article, referring to a dislike of test-first, or of evolutionary design.

            I argue that the code quality and class design you get from that approach to unit-test TDD done right will be the same as TDD with VEST, that the cost of change is lower with VEST, and that VEST (as in, the practice of acceptance testing with full in-memory implementations of slow parts and the other bits and bobs) provides more benefits that are more palpable to users, given the same amount of effort on the testing part. It does not preclude TDD (or ATDD), nor replaces it. It’s a way to architect the code around your testing harness, it’s orthogonal to being TDD or ATDD or BDD or whatever else may be the flavour du jour.

            That said, I strongly believe that mocks, through the use of mocking frameworks, in the way they are used, are dangerous. You may argue that guns don’t kill people, I’d argue that they help tremendously, and that the mock should really be written by whoever knows the target the best, and that’s why I suggest in VEST that you should not write mocks yourself.

            I’m going to see if I can change the entry a bit to clarify the relationship with TDD, I talk about it in the intro and that was a mistake, but I tend not to want to edit heavily something after it caused conversation. It does mean I repeat myself much in the comments, so it may well be that I ought to be writing another entry.

          • http://blog.thecodewhisperer.com J. B. Rainsberger

            I think the “misconception” comes from your calling yourself a recovering TDD addict. I interpreted that as you turning your back on TDD, and this VEST sounds like at least part of your replacement for it. I guess you intended it a different way?

            I like your impulse to frame your tests as far up the call stack as possible. I strongly prefer that outside-in approach, although I do find that sometimes I need to design from the inside-out — risking the Chunnel Problem (example: my API needs data the client doesn’t have) — to wrap my head around what I’m doing, then I smooth over the connexions from the outside in once I’ve learned enough about where I’m going. I’ve observed, however, that a mechanical resistance to smaller, more focused tests, duplicates code and intent in tests, which spreads by copy/paste and creates a worse (more-easily-justified) brittleness than what many people fear of mocks.

            You advocate what I call Lightweight Implementations, things like in-memory databases, over mocks. You even advocate shipping Lightweight Implementations alongside the Industrial-Strength Implementations, which I love. Dynamic mocking libraries came about in response to monstrosities like Java’s ResultSet interface: I want to mock close() for this test, but I have to implement 127 methods to do that. I have to specify the correct behavior for 126 methods that I don’t care about — grossly over-specifying my “givens” — to check that my method closes its connexion. This was not a theoretical issue. :)

            Still, the downside of mocking methods: you can easily destroy the API’s cohesion. You can easily mock related methods in a mutually-incompatible way. Lightweight Implementations address this problem, by nudging the programmer towards less-obviously-garbled implementations. It’s no guarantee, any more than the mocks.

            I’ve done it both ways for years, and I tend to favor mocks, but use Lightweight Implementations when I find the mocks too complicated for a specific situation, especially when I tend to mock the same trio of methods together in 90% of the client’s tests. Of course, what really tends to happen is that the Lightweight Implementation takes over some significant portion of the API, becoming the Industrial-Strength Implementation that delegates some details to a new, smaller interface, which I more easily mock. For example, the database becomes a write-through cache for the Dictionary of data I maintain in memory. Not always, but sometimes. The write-through cache becomes a smaller, simpler API to mock, so that a Lightweight write-through cache becomes a heavyweight solution. (I just want to mock whether I have to call load() this time!)

            I think we understand each other better, and I’m happy for the chance to discuss this with you. Sometimes we internalise the context of our advice so deeply that we forget that not every reader knows that context, and they draw dangerous conclusions. Articles like this one remind me about that danger, and bring a lot of that implicit context out into the light, which I believe needs to happen every so often. :)

            (BTW, I’m Canadian. I say this only because while I believe that guns don’t kill people, too many guns makes it too easy for people to kill people in the heat of the moment, and *that’s* why I want to limit the availability of guns. Same for mocks. I don’t like mocking concrete classes, because it becomes too easy to mock an object without noticing the design improvement that that difficult-to-write test is trying to scream at the programmer. I want the programmer to have to think about the effects of extracting an interface. I especially want the programmer to think “How stupid!” and then talk emself into it.)

            (BTW, BTW: “emself” is a Spivak pronoun, not a typo.)

          • http://codebetter.com/sebastienlambla/ Sebastien Lambla

            Thanks for the feedback, it highlights deficiencies in my blog entry. Time for a new draft. :)

  • David

    One of the purposes of unit testing is to make it easy for other developers to modify your code. This makes the internal “contractual” details important – which is why heavily unit tested code so important.

    Your idea of making functional or integration tests so fast that they can be run as frequently as unit tests is quite intriguing. But I see it as a complementary capability not a replacement.

    • http://codebetter.com/sebastienlambla/ Sebastien Lambla

      It depends on what you consider users. I consider people that are likely to change your code to be a different breed from devopers using your code. The investment is not the same, and what is efficient for users is not necessarily useful for people with which you share a common understanding and knowledge. For the latter, code craft, self descriptiveness properties are usually sufficient. It’s not to say unit testing as a documentation tool is not useful, it is to say that such an approach has failed to catch up by far and large and that time can be better used approaching the problem differently.

  • http://www.aquabirdconsulting.com/ Khalid Abuhakmeh

    So you explained the Why, and you explained the What, but is there going to be a How?

    • http://codebetter.com/sebastienlambla/ Sebastien Lambla

      As I said above, I’ve not covered samples. It would be a natural evolution for a future blog post indeed. :)

      • http://www.aquabirdconsulting.com/ Khalid Abuhakmeh

        I don’t know if this is the same, but I found that using storage engines that let you use in memory versions of themselves provide a much more honest test (all be it slower sometimes). I think the holy grail of a test would be to spin up you whole app in a “sandbox thread” and run it.

        • http://codebetter.com/sebastienlambla/ Sebastien Lambla

          That would be indeed a good target for a fully VESTed system :)

  • Christian Rondeau

    So in short, the idea is that companies ship their in-memory software equivalent, rather than everybody having to write their own faulty, hard-to-maintain mocks? That’s a brilliant and simple idea that we will begin offering as a value-add for our API users! I would say this is more about mocking than TDD however 😛

    • http://codebetter.com/sebastienlambla/ Sebastien Lambla

      Those are effects: the shipping of the test doubles, the shipping of test harnesses for those componenets users are expected to replace, are practical results for applying VEST. Focusing tests in top-down fashion and focus at the closest level possible a user will have access to the system under test, while replacing only those parts that are not under control, that’s the reason. :)

      • http://blog.thecodewhisperer.com J. B. Rainsberger

        “…the shipping of test harnesses for those components users are expected to replace…” You’re describing contract tests, although with a phrase that more people are likely to immediately grok. I, too, wish everyone would ship contract tests along with their pluggable/replaceable services.

        • http://codebetter.com/sebastienlambla/ Sebastien Lambla

          Any interaction with an external component follows a contract. I’m a bit concerned of using “contracts” in the .net world as people automatically make assumptions that the two are one and the same concept. Contractual behavior may cover it more clearly?

          • Guillaume L

            I like the concept of a “protocol” that defines how conversation is supposed to happen between consumers and your object (or a substitute thereof).
            By the way, if you’re referring to Code Contracts as the thing .net people will spontaneously picture, I’d argue these are much in the same spirit as the tests you mention or Joe’s Contract Tests, only more coercive. You can ship Code Contracts along with your code and they can also be applied to interfaces, which pretty much covers the “shipped test double” approach you’re describing. It’s still a great idea, I’d like to see more of this in whatever form – and will certainly try to do more of it myself.


    What is the difference with integration test ?

    • http://codebetter.com/sebastienlambla/ Sebastien Lambla

      What is an integration test?

      • Rémi BOURGAREL

        Basically it’s what you’re talking about : instead of testing a unit of your program you try a whole business case.

        • http://blog.thecodewhisperer.com J. B. Rainsberger

          I prefer to distinguish “integration test” from “integrated test”.

          Integrated test: a test whose pass/fail depends on the behavior of many distinct bits of behavior. Not a very precise definition, but that’s a feature, not a mistake. :)

          Integration test: a test to check the integration point (or surface) between Our Stuff and Their Stuff.

          This is exactly why I changed my articles to say that *integrated* tests are a scam, and not integration tests. :)