Ordering Code Construction Tasks

I spent a couple
years as an engineer on large petrochemical construction
projects.  A lot of the mental work in a large construction
project is determining the order of construction for the various
subsystems and physical structures of the plant.  In construction
that order is more or less governed by physical realities and it’s a
well understood problem domain.  In the world of software
development determining the order of coding construction is more
nebulous because we’re not bound by physical laws.  All the same though, the success of our projects is influenced by the order in which we code the pieces of the project. 

 

Framework Fool’s Gold

 

I’ll freely admit it; I’m a recovering “framework-aholic” that’s had some bouts with “pattern-itis” as well.  That’s
what makes me qualified to say that one of the worst project
anti-patterns in software development is building an elaborate
framework first.  It’s a seductive path.  On
one hand there’s the whispered belief that if we can just get the core
framework right, building new services/screens/controllers/applications
will be easy, right?  On the other hand,
it’s just plain fun to exert your creativity to the maximum to build
the ultimate metadata-driven, pluggable framework so new features can
be added without any new code. 

 

There are a couple little problems with building frameworks upfront.  The main problem is that it very rarely works.  You’re essentially guessing what the framework needs to do, and even the best architects can easily guess wrong.  Even
if you’re the best architect in the world, the folks feeding you the
requirements are always changing their minds or coming to a different
understanding of their needs.  Building an all-encompassing framework outside the scope of a specific business project almost always results in wasted effort.  The worst case scenario is that you’re framework won’t deliver any value and results in a lot of rework.  The
best case scenario is that your framework does work, but you don’t reap
much early feedback or have any opportunities to deploy the system
early.

 

Too much abstraction with an overarching framework early on can make an application viscous.  Uncle Bob describes design viscosity as a malady that makes a design harder to preserve as changes are made.  If
it’s easier to make a change by going around a framework or drilling
exception cases into a framework with “if/then” statements, you’ve
created a viscous framework that’s might be causing you more harm than
good.  If you start to hear the phrases
“can’t we just go around the framework?” or “this would be easier
without the framework” or even worse “we need to get [Some Guy] to
change the framework first” you’ll know that you’re going down a
painful path with your framework.

 

The worst is when the
framework turns into the mutant plant from the Little Shop of Horrors
as it takes on a perverted life of its own.  “Feed me Architect, Feed me!”

 

To give you an
anecdote that’ll get me in trouble when they read this, a part of my
team this past spring was embarking on an overly elaborate,
future-proofed architecture for our product rewrite.  They
were having fits debugging and testing the application in large part
due to some unnecessary technical infrastructure (and severe
performance problems down the road).  About halfway through the project they decided to simplify their architecture.  The result was what they call the “Great Refactoring of Aught Five.”  Funny name aside, there’s no such thing as a big refactoring, only rewrites.  They
would have clearly been better off starting simple without the
abstractions and infrastructure (plus there’s still some dead code
detritus lurking in the codebase we’ve got to kill off some day). 

 

I proscribe (now) to
the view that you should harvest a framework from an application only
after it has been proven to be useful or necessary.  When you do build a framework, don’t try to create something grander than what you need right now.  I’d
also say that it’s much, much better to “dogfood” a new framework as
part of a project rather than trying to write a “Hammer without a Nail”
framework that doesn’t really fit the needs of any project. 

 

Slice the Work Vertically, Not Horizontally

 

The metaphor of software development as construction is common, but horribly wrong.  Approaching
software development as constructing a structure on top of a
foundation, then adding the finishing trim can be a slow, painful path
to project failure.

Time and time again I’ve learned or observed that projects go much smoother when you build vertically instead of horizontally.  What
I mean by this is that you build a new system by creating a feature at
a time with the entire stack of UI, business logic, and database work
to make the feature fully functional.  As you build new features you religiously eliminate duplication to avoid creating a Stovepipe anti-pattern and harvest reusable code.

 

One of the more disappointing project failures I’ve observed was the very first .Net pilot project in a VB6 shop.  I
wasn’t involved with the project, but I had a stake in the project as
it was going to prove the viability of .Net for application development
and also be a pilot for doing iterative development.*  Unfortunately
they blundered by crafting an iteration plan that called for them to
write the entire data access layer first, the business layer second,
then finally put together the user interface.  It failed miserably.  When the project was cancelled midway they didn’t have the slightest bit of deployable code because they had focused solely on the backend database.  One
of the other things they found out was that the data access they coded
wasn’t exactly what the business and UI layers needed.  In other words, they had wasted effort by speculatively creating code that turned out to be unnecessary. 

 

One of the best, positive lessons I’ve learned working on Agile projects is the importance of getting feedback early.  You
can’t get much usable feedback from the end user when all you have to
show is database diagrams or UML class diagrams (a word of advice,
don’t ever show a customer a UML diagram unless it’s some kind of
experiment in hypnosis).  Besides, you know damn well that the thing most likely to change is the user interface.  You’ll also understand any piece of the system better when you’ve built the entire chain of calls.  You simply cannot know any piece of code is done until its clients are complete.

 

I did a project about
3 years ago where I created an elaborate, overly generalized workflow
framework first before creating any working user interface.  The framework worked fine and the system still humming along, but the lack of a UI became a problem.  The
other members of the team didn’t truly understand the point of the
framework because they didn’t have a set of screens to see the concrete
usage of the framework.  The users and the project manager had no idea what our progress was because they couldn’t see anything working.  We didn’t get any usability feedback from our user base until dangerously late in the project.  Yeah, you can claim earned hours for the code you create, but what value does that really represent?  The
Agile methods have gotten it exactly right by tracking progress by user
stories completed, and defining that “done, done, done” for each user
story requires customer approval.  And besides, the customer probably only counts a working user interface as progress anyway.

 

One of the desirable
qualities of an iterative process is being able to deploy a partially
complete system if the project has to come to an unexpected stop due to
changing priorities or funding problems.  The only way to accomplish that goal is to build an application feature by feature rather than layer by layer.

 

Client First

 

Earlier this year
there was a raging argument on the web on the merits of writing web
services “Code First” (generate WSDL from code) versus “Contract First”
(generate code from WSDL)** that I mostly ignored.  I
think there’s a time and place for both approaches, but the one
constant is to write (or at least simulate) the client first whenever
possible.  The requirements and the signature of any service will be largely determined by the needs of its clients.  “Client First” doesn’t just pertain to web services, it useful for any class that’s a service provider to other classes.  If
you’re doing Test Driven Development (or not I suppose), you might use
a mock object or a stub to be a stand in for a service while
you concentrate on creating the service client.   Once they’re completed, the unit tests for the service client will very accurately define the requirements for the service.  For the obvious example, don’t write data access or persistence code long before the business logic that needs to be persisted.  I
know some SOA enthusiasts who have a “build it and they will come”
mentality to creating web services left and right, but what’s the use
of building something that nobody wants?

 

Configuration Last

 

When it is time to
build some sort of configurable framework, my strongly worded advice is
to concentrate on the classes that do work first, and worry about the
configuration strategy second.  I read that years ago in Craig Larman’s excellent book on design patterns and I clearly see the wisdom in that advice.  StructureMap is a working dependency injection tool, but it started its life as the world’s greatest ORM tool.  I got bogged down obsessing with the configuration infrastructure and never wrote a single line of actual O/R mapping code.  Only later on did I take the abandoned configuration code and turn it into a dependency injection tool to use on a project.  I still wonder what would have happened if I’d been able to release a solid O/R mapping tool for .Net 3 years ago.

 

Besides, I think
StructureMap and the other open source IoC/DI tools make it
ridiculously easy to create configurable plugin strategies compared to
what it took 3 or 4 years ago.  Just push all
configuration information and dependencies in through constructors
or setter properties and let the IoC/DI tool put things together for
you later.

 

 

* This is a terrible example of working iteratively.  The idea of working iteratively and incrementally is to build fully working pieces of the application, one at a time.  What they did was just BDUF + Waterfall with some trappings of iterative development.

 

** I
had a colleague in my old architecture team who thought all of us
should be able to write WSDL by hand as a qualification for being on
the team.  I blew him off while the others nodded fervently.  There’s
a long list of holes in my resume I wish I had more familiarity with,
but writing WSDL by hand isn’t on that list at all.

About Jeremy Miller

Jeremy is the Chief Software Architect at Dovetail Software, the coolest ISV in Austin. Jeremy began his IT career writing "Shadow IT" applications to automate his engineering documentation, then wandered into software development because it looked like more fun. Jeremy is the author of the open source StructureMap tool for Dependency Injection with .Net, StoryTeller for supercharged acceptance testing in .Net, and one of the principal developers behind FubuMVC. Jeremy's thoughts on all things software can be found at The Shade Tree Developer at http://codebetter.com/jeremymiller.
This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Rooney

    “I blew him off while the others nodded fervently.”

    I guess this means something different in the you ess of a

  • flipdoubt

    The PDF on the other side of the “design viscosity” link is broken. Can someone point to a valid copy of that PDF?

  • http://www.dotnettricks.com/ Fregas

    Loved your article, and it matches my job experiences precisely. At my second .NET job, the developer that was supposedly my senior wrote all his stored procs, then all his DAL, then all his Business layer, and so forth. There was so much code that never got used because the framework was wrong.

    Someone mentioned that the vertical approach has downsides as well. I’d say the downsides to the horizontal approach vastly outweigh the downides in the vertical. And the vertical does assume you build all three layers: just build them for one feature at a time, not all of them at once.

    Good job Jeremy. i’ll probably blog about this as well.

  • Schroeder

    Funny you should mention that. I’ve been beating my head against a wall for the last two hours trying to get our company’s framework to execute a DataReader with some inline sql. It would take a grand total of 2 minutes to code it up outside of the framework and it would work. Argh.

    Consequently, the ExecuteDataReader command isn’t even exposed via the Framework so I have to do some fancy footwork to even get down to that level. I mean, who would want to use a DataReader anyway, right? We might as well just throw even the most trivial resultset into a full-fledged DataSet.

  • http://weblogs.asp.net/pgielens/ Paul Gielens

    Hi Jeremy,

    I wrote a few words on my (similar) experiences here:
    http://weblogs.asp.net/pgielens/archive/2005/11/27/431604.aspx

    Best Regards,
    Paul Gielens
    http://weblogs.asp.net/pgielens/

  • http://weblogs.asp.net/fbouma Frans Bouma

    Your article started out great until you reached the point where you started to give out advice about client, ui etc.

    What I miss in your whole approach is solid computer science based software engineering. The ‘this is how to do it’ part of the article breaths ad-hoc ‘I learned this in the field’ experience. Let me explain. The problem you discuss with horizontal development is there, and you describe it perfectly. What you fail to add to your analysis and conclusion to use a vertical approach is that the vertical approach also has downsides, one being too UI focussed. Countless projects failed miserably because too much time was spent on the UI while the foundation of the software was nowhere in sight. This happens because to be able to do vertical development you first have to have 3 tiers working, and a somewhat working ui.

    Nevertheless, I miss functional research, I miss analysis to come to a subset of functionality (!) to implement, I miss technical research of that subset to come to a set of technical subsystems to implement, I miss discussion about those solid, CS, proven parts of software engineering which are the base of WHAT you have to build (the functionality) and HOW you have to do that and above all….. WHY you have to build it that way.

    functional research and the other research elements of the first half of the project are done together with the client, to get a clear picture what has to be done. Once that picture is there, you can write the code. The beauty of it is that you have a theoretical base for what you’re doing: from code to technical research to functional research so you can go back to its roots why that given piece of code is implemented and vice versa: where is this piece of functionality implemented.

  • http://www.dotnetjunkies.com/WebLog/chrisdonnan Chris Donnan

    This is a great article. You have been on my feed list for a few weeks now. Your section “Slice the Work Vertically, Not Horizontally” is directly related to a battle I have been having with a client lately…. see my full commentary on this blog entry here ….

    http://www.dotnetjunkies.com/WebLog/chrisdonnan/archive/2005/11/27/134012.aspx

    Thanks and keep up the interesting bloggin’
    -Chris