Leaky Abstractions and the Last Responsible Moment for Design

If you haven’t read it before, go check out Joel’s Law of Leaky Abstractions.   I bumped into some code in one of our projects last week that is a classic case of a leaky abstraction.  It looked something like this:

 

            public DataTransferObject ProcessTheMessage(Message message)

            {

                  IMessageHandler[] handlers = createHandlersForMessage(message);

                  DataTransferObject dto = new DataTransferObject();

 

                  foreach (IMessageHandler handler in handlers)

                  {

                        // Check for special case

                        if (handler is SpecialMessageHandler)

                        {

                              // Downcast to the specific MessageHandler

                              SpecialMessageHandler special = (SpecialMessageHandler) handler;

 

                              // Check if the message is from our BigClient with special

                              // “needs” – Our code didn’t do this, but I’ve seen this

                              // far too frequently

                              if (message.ClientId = “BigClient”)

                              {

                                    special.SendEmail(message);

                              }

                              else

                              {

                                    special.Audit(message.Header, message.ClientId);

                              }

                        }

                        else

                        {

                              handler.Process(message, dto);

                        }

 

                        // Another exception case if the MessageHandler is Security

                        if (handler is SecurityMessageHandler)

                        {

                              if (!(SecurityMessageHandler).Authorized)

                              {

                                    dto.Success = false;

                                    break;

                              }

                        }

                  }

 

                  return dto;

            }

 

There is a generalized array of IMessageHandler objects that take their turn processing a Message object, or at least that’s the way it started.  At some point the developers needed to code a special case to short circuit the message handling chain if a specific handler was encountered, or a specific client.  For whatever reasons, the developers chose to go outside the convention of the existing IMessageHandler abstraction.  Once the abstraction springs a leak with one exception case, more exception cases are sure to follow rapidly.  A leaky abstraction is a very clear sign that the overall structure and design of an application is succumbing to entropy and heading for its end of life as changes to the system become too risky or costly.

 

 

Preventing Leaky Abstractions

 

I’d say that there are a couple different causes of abstractions springing a leak.  System design and architecture has to be socialized throughout the development team to be successful.  I think there is a “this is the way we do this” factor that has to be in place within a development team for any abstraction to truly work.  Developers that don’t understand the purpose and usage of an abstraction will go around it anytime they’re under schedule pressure – i.e. almost always.

 

The second big cause is abstracting too early.  When you make a design decision too early you are speculating without all the facts.  To some degree I think very experienced developers and architects can accurately predict what will be useful, but it’s still a gamble.  When you reach too far ahead in your architecture you incur a risk of creating unnecessary infrastructure or creating an unnecessary amount of complexity.  It’s taken me a couple of years now but I think I finally buy (most of the way) into the XP idea of YAGNI.  Abstracting too early is acerbated by changes in requirements or priorities.  The facts and understanding of the business process that underpinned your abstraction have changed and the abstracted model is busted.  Wait to introduce the abstraction until it’s necessary or valuable.  More on this subject below.

 

Architect Hubris and the Siren Call of Metadata-Driven Architectures

 

One of the biggest, most colossal mistakes a software architect can make is that of hubris.  “Architect Hubris” is the belief that the architect(s) can design and code an application framework upfront that will reduce the remaining development to a mere exercise in configuration.  I came awfully close one time with a workflow system that was almost completely configurable – via no less than 65 database tables!  The “now its just configuration” moment never arrived.  There was always some kind of exception case that didn’t quite fit into the abstracted model.  It’s really too bad that we spent so much time getting the workflow framework upfront instead of just coding what the business asked for in specific.  We delivered what we had to, but there was a lot of wasted effort in architectural infrastructure that provided no real business benefit.

 

Favor Composition over Inheritance

 

I think metadata driven design and extensibility mechanisms still has its place, but I use it differently today than I did back then.  Where I extensibility I do it by abstracting at coarse-grained interfaces, not abstract classes.  I then use StructureMap (shameless plug) to configure object graphs.  I never assume anymore that all implementations of an interface can inherit from a single superclass or be described by an identical set of configured metadata.  Read more from Erich Gamma about the “Favor Composition over Inheritance” principle.

 

The Last Responsible Moment

 

I got to participate in a panel on TDD in Austin with Mary Poppendieck last fall.  She spent quite a bit of time talking about the value in making decisions at the last responsible moment.  The key is to make decisions as late as you can responsibly wait because that is the point at which you have the most information on which to base the decision.  In software design it means you forgo creating generalized solutions or class structures until you know that they’re justified or necessary. 

 

Abstracting too early is problematic, but waiting too long to introduce an abstraction is almost as bad.  You either end up living with duplication in the code (bad) or pay a cost in a larger refactoring to introduce an abstraction to remove the duplication.  Continuous design as practiced by Agilists is not another flavor of chaotic “code n’fix.”  It’s design all the time.  You have to be constantly scanning your code and design and looking for code smells creeping in and opportunities to create abstraction that remove duplication or create greater efficiencies.

 

I’ve worked with some people who interpret Agile development to mean that you never do any design activity outside of coding.  Let me say this as clearly and forcefully as possible, always be thinking ahead!  Consider different abstractions, research design patterns to reduce conditional logic, and look for reoccurring patterns in the code that can be generalized.  Just don’t implement anything more than you have to for the functionality at hand.  Large refactorings shouldn’t sneak up on you.  You should always have a list of possible improvements to your design floating around both in your head and in conversations with your teammates. 

About Jeremy Miller

Jeremy is the Chief Software Architect at Dovetail Software, the coolest ISV in Austin. Jeremy began his IT career writing "Shadow IT" applications to automate his engineering documentation, then wandered into software development because it looked like more fun. Jeremy is the author of the open source StructureMap tool for Dependency Injection with .Net, StoryTeller for supercharged acceptance testing in .Net, and one of the principal developers behind FubuMVC. Jeremy's thoughts on all things software can be found at The Shade Tree Developer at http://codebetter.com/jeremymiller.
This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Paul

    Nothing to say except great article.

  • Bernie Woolfrey

    (Wow… fast turnaround. Don’t you Yanks sleep?)

    Open, flexible design, good OO and loose coupling help, and for simple stories, we’re avoiding the sort of churn you mean.

    Good design practices make dealing with discovery requirement detail possible, not easy.

    It’s mainly where complex business rules and processes are involved.
    I think that at root, our difficulties are caused by failing to completely capture and understand the details of complex rules and processes. The penalty involves code and database re-design and re-implementation: Potentially heavyweight activities that Agile proponents describe by the lightweight name of “refactoring”.
    Waterfall development, although it has many defects, attempts to formalize the analysis and documentation of business processes and rules. The design phase then is easy(er).
    Of course we may just end up designing and implementing something that utterly fails the end-user’s needs.

    I’m conscious of this potentially wandering a little off-topic because what initially caught my attention was the title of this blog. Dealing with the iterative nature of requirements discovery is one facet of the challenges we’re facing putting Agile into practice. I guess the reality is that we will NEVER be able to get the whole picture before we begin design. Using a flexible methodology and a flexible design and implementation recognises this reality.

  • http://codebetter.com/blogs/jeremy.miller Jeremy D. Miller

    Ok Bernie,

    “How can you expect to create a good design before the details of (at least most) requirements have been discovered?”

    You design ONLY what you need for the requirements you’re working on right now. You do know, or should know, the requirements for the current iteration. Iterative development doesn’t mean endless design churn. If there is, you’re doing something wrong.

    - Keep it simple upfront. No elaborate abstractions until there’s an obvious need.

    - Pay very, very close attention to good OO design in every single iteration. Highly cohesive classes with loose coupling promotes evolutionary design. Classes with low cohesion and coupling between classes is generally what will make the kind of design churn you’re worried about much worse.

    - Follow the Open Closed Principle and the Single Responsibility Principle. That maximizes your ability to do evolutionary design.

  • Bernie Woolfrey

    This is very timely and interesting.
    Design is one of the many problems I’m trying to resolve as our team attempts to apply Agile techniques:
    How can you expect to create a good design before the details of (at least most) requirements have been discovered?
    When the Leaky Abstraction appears, it probably means that a previously unknown requirement has surfaced.
    I guess that most Agile exponents will say “refactor”, but that cna be expensive (both in time and in spirit). And what does the PM say when you tell them for the nth time that “We’ll have to refactor the UI/Business Facade/Database/whatever..”?
    Designing (and building) from a user story like: “As a user, I want to be able to submit and validate a data file” is a death sentence for whatever gets done in its first iteration, and the idea of using an iteration as a tool for eliciting requirement detail seems plain wasteful.

  • jmiller

    Hold on Aliostad,

    - Composition based structures in some circumstances lead to more potential reuse than inheritance
    - TDD w/o OO in C#/VB.Net/Java doesn’t work all that great, so TDD purism often leads to OOP purism
    - Favoring composition over inheritance for OOP is an old, old piece of advice that long predates the rise of TDD
    - XP (or RUP or whatever) doesn’t lead to bad designs, bad designers lead to bad designs

  • aliostad

    Great stuff Jeremy like the rest.

    I have one issue over interface vs. inheritance (I take composition over inheritance synonymous to it) that you do not get code reuse with interfaces.

    For what it is worth, I would use inheritance for code re-use and interface for infra-structure or service re-use as well as loose-coupling. So internally a subsystem or service can use inheritance to help with code reuse (behaviour reuse) but for integration or loose-couling I will only expose interfaces.

    I find it a bit too shallow that we sometimes we forget what we have achieved by using an approach that is not fashionable anymore. Nowadays TDD and XP are good and OO is bad. OO works what does not work is OO purism. And to be honest I have now -thanks God – lived enough to see rise and fall of all many approaches when it is taken to the level of purism. It applies pretty much to everything, XP purism, TDD purism, interface-based programming purism – maybe it is too early for some of them.

    I have already seenadverse effects of using XP. It is bad, rushy design. I would personally design for tomorrow – and not the day after tomorrow or next week – and implement for today.

  • http://volzsoftware.com jdvolz

    This is an excellent article that brings up several points in way that I had not previously envisioned them.

    While I agree that one should not design the software before the requirements are well known, it is also irresponsible to jump into a project without some plan as to how you are going to handle change, or the introduction of new abstractions. If you enter coding without a plan, and then deny your coders the chance to create helpful abstractions because it “will take too much time” then you are headed for a disaster.