Jeremy D. Miller -- The Shade Tree Developer

February 2007 - Posts

• Lipstick on a Pig

  I'm a card carrying member of the SOA skeptics club, but I'm going to push for a dose of SOA thinking to my current client.  I do understand the potential benefits of things like BAM, BPM, and flexible orchestration if the entire enterprise were really Service Oriented, but I generally think of SOA as a kind of fool's gold for ivory tower integration architects who desperately want to play God without ever getting their hands dirty (my advice for these folks is to go get a copy of Civ IV).  For the most part, the SOA initiatives I've seen have only led to extra complexity and reduced productivity in development without ever coming close to the orchestration nirvana envisioned by the architects.  As I see it, the positive goal of a Service Oriented Architecture is to eliminate duplication of effort in software projects across the enterprise by reusing existing code assets and reducing the cost of changing the behavior of software to keep pace with evolving business needs.  As I've stated in other posts, it's usually easier and generally less risky to create new behavior by writing brand new code instead of breaking into existing code to make modifications.  Call me Pollyanna, but I firmly believe that those very goals can often be achieved at a lower cost by a relentless focus on creating well-factored code backed up by a healthy dollop of test and build automation (SOA certainly doesn't abrogate the need for good code behind the service points, but that's a gripe for another day).  Aw, but here's the rub:  the mass majority of the code out there in production is of dubious quality and the kind of test and build automation that enables change simply doesn't exist for most systems.  So back to the usefulness and appropriateness of an SOA solution, there's a phrase I've often heard or used in regards to exposing services to a nasty legacy codebase to try to extend the usefulness of said legacy code that's particularly applicable to my current project -- "Lipstick on a Pig."  This piggie needs some lipstick Here's the scenario, you're building a new client application or work process automation solution that needs to integrate or use an existing legacy server application with oodles of business logic and functionality.  What can you do?  I can think of three options off the top of my head: Write the new client as part of the legacy system.  In our case the legacy system is in a niche language, and they are committed (justifiably so IMHO) to producing all new clients in either WinForms or Swing.  This option is not on the table. Reproduce the necessary functionality in the new application.  This is the approach that we ultimately were forced to choose, but nobody is, or should be, happy with this approach.  Reproducing the existing functionality made the project take much longer, and add considerably more risk to the project, than was envisioned when the project was launched.  I say risk because we're still not exactly sure what the correct business logic should be because the original code is not entirely readable or documented.  We have also increased my client's maintenance costs considerably by duplicating functionality without retiring the existing legacy code.  Anytime this functionality needs to change, they'll have to change and test both sets of code.  Because all of the integration happens through a shared transactional database, the implicitly tight coupling between our new code and the existing code is not obvious. Find a way to use the existing legacy code in place as it is.  In effect, they could desperately use a service layer of some sort between their legacy code and newer client applications to avoid the duplication.  In terms of design, decoupling the legacy code enough to slide in a service layer is going to be difficult (I'm betting most of the functionality is in the 2-tier UI clients), but in the end I think it'll provide a heck of a lot more value to them.  Once the service layer is in place, new development work should get easier because they can just hook right up to the services with a well-defined contract.  Even better yet, the service layer should provide the ability to go to work on the service backend and start moving code over to a better factored codebase in a mainstream platform.  Putting lipstick on this pig will generate business value. This piggie needs to go home Some of my attitude on SOA originates from my days inside internal IT in a Fortune 100 manufacturing company that really is a quite logical fit for a rational SOA approach.  On a particular project, we were tasked with making significant improvements to the nasty, failure prone shipping system (as in "shuts down the factory lines when it goes down).  A senior enterprise architect was given the responsibility for determining the overall architectural changes.  His entire plan pretty well came down to replacing the existing integration points with webMethods.  The point of his strategy was to decouple the clients of the nasty legacy application from the details of the legacy application first, then make improvements to the server system at some later point.  He was perfectly honest in describing his plans as "Lipstick on a Pig."  I've always been primarily an applications guy, so I was flipping out that we were spending a lot of energy that wouldn't result in a single, observable improvement in terms of uptime, performance, functionality, or reduced support costs.  Worst was the fact that the business was told they were paying for overdue structural improvements to eliminate downtime incidents.  I even expressed an opinion that the new webMethods integration infrastructure would only add network chatter to an already chatty system and further degrade the already deficient performance.  To this day, I don't think anything far short of a full rewrite would have represented a significant improvement (and they did replace it after I was gone).  Lipstick wasn't going to help this pig at all.   And no, I don't necessarily think SOA is a bad thing, but done badly or unnecessarily, SOA is a blight upon the land.     Posted Feb 24 2007, 05:31 PM by Jeremy D. Miller with 17 comment(s) Filed under: Legacy Code, Maintainability

• Don't Let the Database Dictate Your Object Model

  Before you go on, I'm specifically worried at the moment about "Logic Intensive Systems" here.  Systems that perform complex calculations, make optimizations, determinations, decisions, etc.  Many, if not most, enterprise applications have both an object model and the database model.  For the most part, convergent evolution will probably lead the two models to be very similar, but it's potentially dangerous to constrain the two models to match perfectly because the two models reflect different concerns altogether. Database Model - When you design a database model you're primarily worried about the best way to structure data for efficient storage and retrieval, while also enforcing data integrity rules Object Model - The object model is first and foremost concerned with modelling the behavior and business logic of the system. Ideally, I'd like to work on these two models somewhat independently and allow both models to reflect their different concerns first, and each other second. O/R mapping of all flavors isn't that difficult to use (authoring an O/R mapper is a totally different story) when the database and the object model are very similar.  The problem is that making the persistence easier by locking the object model to the database model can make writing and consuming the business logic harder.  I'm working with a system that uses business objects that are basically codegen'd one to one from a legacy codebase with 400+ tables.  The temptation and driver for codegen'ing the business objects is obvious (400+ tables).  The problem I'm seeing though is that consuming these business objects in the service layer is that the business objects do not really reflect the behavior of the business logic.  Even worse in my mind is the fact that there is no encapsulation of the raw database structure from the service layer.  If we had designed the business classes to reflect the behavioral needs to make writing the business logic easier (and testable), we would have ended up with a quite different structure.  Just to throw up some examples: Big tables don't map to a single object.  I don't think it's possible that a class with a 100 different properties can possibly be cohesive.  We'd be much better off in terms of writing business logic if that 100 column table is modelled in the middle tier by a half dozen classes, each with a cohesive responsibility.  It may make perfect sense to have only one table for the entire object hierarchy, but big classes are almost always a bad thing. Data Clump and Primitive Obsession code smells.  A database row is naturally flat.  I want to do a bigger post on this later, but think about a database table(s) with lot's of something_currency/something_amount combinations.  There's a separate object for Money wanting to come out.  If you make your business objects pure representations of the database you could easily end up with a large amount of duplicate logic around currency and quantity conversions. Natural cases for polymorphism in your object model.  I think the roughest part of O/R mapping is handling polymorphism inside the database.  Check out Fowler's patterns on database mappings for inheritance. Back to the 400+ table problem.  Yes, their database model is huge, but they aren't actually consuming most of the generated classes anyway.  Looking at the bigger picture, I think it would have probably been easier to deal with the dissonance between business logic needs and the existing database structure in the database mapping, even though that potentially represents more work to do persistence,  instead of exposing the raw database structure to the business domain classes and their consumers.  In this case, I think making the business logic easier to use and consume would more than offset the extra persistence cost.  Since I would expect the business logic to change more often than the database structure, I would also prefer to optimize my ability to modify the business logic in isolation from the database. Besides, you definitely want to minimize coupling to a legacy database on the off chance that you might get to fix it up or move away from it later.    WARNING -- If you are going to use O/R mapping of any kind, it's even more important than ever to properly enforce referential integrity rules in your database.  I don't know if it's just my bad luck or what, but the legacy databases I've hit in the past couple years were all missing a lot of logical referential checks --and unexpected problems with orphan records quite logically ensued. Object Relational Mapping is Hard at the Edges If you haven't already read it, take a look at Ted Neward's seminal paper on the O/R quagmire.  I thought he was exagerrating the problem on my first read, but now I'm not so sure.  Automated, metadata driven O/R mapping (and I'm broadly including the LINQ varieties and codegen tools here too) gets really nasty at edge cases.  There comes a point when the metadata driven approach starts to hurt more than it helps and it's probably better to revert to hand-rolled database mapper code in these cases.  This issue is part of what drove me to write Being afraid of your backhand.  One way or another, you will occasionally need the ability to allow the object model to diverge from the database model. Not to put words into Neward's mouth, but reverse engineering your business domain classes from an existing database definitely fits his analogy comparing O/R mapping to a quagmire.  This especially holds true for a legacy database that isn't, shall we say, pristine in structure. Heck, the last time I willingly wrote a stored procedure was to take advantage of a PL/SQL feature to pull a logical hierarchy of data out of a flat database table.  It worked beautifully thank you (of course the rest of the team threw a fit about using a sproc).  The Role of the Database When people think about the role of a database in an enterprise system, I think there's a spectrum of thought with two polar extremes.  I think the proper place in the spectrum should vary by application, but we all come in with presuppositions on the best way to write software based on our prior experences that impact the direction of our design.  Where you sit on this spectrum has a lot to do with how you will approach application architecture vis a vis the database: The database is paramount, and the system is expressed and understood in terms of the tables and rows in the database.  The application code and even user interface is just a conduit to get information back and forth into the database.  You design the database first and then build the business and data layers to match the database.  In the .Net world we might just consume raw DataSet's in the application, effectively just working with the database tables offline. The behavior of the system, primarily in the middle tier and user interface, is paramount, and the database is "just" a means to persist the state of the system.  The database is either built to match the business classes or designed somewhat independently. Reporting applications and simpler data entry applications can happily sit at the #1 data centric end of the spectrum.  I think any system with significant business logic really needs to be edging over to the second end of the spectrum.  The tricky part is recognizing when an application crosses the line from purely data centric to logic centric.  I'm of the opinion that applying data centric development approaches to logic intensive systems leads to a world of trouble. Posted Feb 23 2007, 09:18 AM by Jeremy D. Miller with 55 comment(s) Filed under: Legacy Code, Database and Persistence

• Being afraid of your backhand

  I haven't played Tennis in several years, but when I did I had a truly awful backhand.  I would routinely kill myself trying to run across the court to position myself to hit a forehand. I'm bringing up my Tennis defficiencies as an analogy for development.  I'm seeing too many cases where teams are just killing themselves coming up with one size fits all abstractions or elaborate MDA-ish code generation strategies to avoid, God forbid, writing one-off code that only satisfies the immediate needs.   Just like me in Tennis, teams are putting themselves in unnecessarily difficult positions because they're too afraid of their backhand (just writing simple code).  I suppose there's an occasional need for large scale codegen and abstraction is definitely important, but sometimes you just need to hit the backhand and write simple, one-off code.   I think you can almost break coding philosophies into two general camps. Camp #1:  Coding is too hard, so let's not write code anymore.  Model Driven Architecture, Executable UML, Business Rules engines, Rapid Application Development Camp #2:  Coding is too hard, so let's make coding easier and more productive.  Refactoring tools, dynamic languages, TDD, Continuous Integration, Ruby on Rails, etc. Put me in Camp #2.  I've also got a theory that Camp #1 is scarred C++ vets or VB guys and Camp #2 is ex-Smalltalkers and the like. Posted Feb 21 2007, 09:55 AM by Jeremy D. Miller with 21 comment(s) Filed under: Ranting

• Preview of StructureMap's new Configuration DSL

  Mind you, this is strictly a preview.  In fact, I'm hoping to incorporate feedback from this post to improve usability.  I'm adding a new Fluent Interface (think RhinoMocks) DSL to StructureMap vNext for programmatic configuration and wiring.  I see it as an excuse to play with Fluent Interface coding an easier alternative to the attributes or Xml configuration for simpler systems.  I don't necessarily see the DSL as much easier to use than the Xml or Attributes, but I think it is potentially much easier to read and maintain.  The details aren't locked in yet, but you will be able to both bootstrap the StructureMap configuration at application startup and also embed "Registry" classes in assemblies that will be picked up at runtime.  In this release I'm trying to make it much easier to manage StructureMap configuration across multiple assemblies and subsystems for composite applications.    Adding a new Instance Here are a couple samples of defining new instances in memory.  If you've used StructureMap in the past, notice that you do NOT have to specify the PluginFamily/Plugin configuration separately.  That's now taken care of behind the scenes for easier configuration. For reference, the constructor for the WidgetRule class looks like this:     public class WidgetRule : Rule     {         private readonly IWidget _widget;           public WidgetRule(IWidget widget)         {             _widget = widget;         }             public IWidget Widget         {             get { return _widget; }         }     } And here is some samples from the unit tests:             // Add an instance with properties             registry.AddInstanceOf<IWidget>()                 .WithName("DarkGreen")                 .UsingConcreteType<ColorWidget>()                 .WithProperty("Color").EqualTo("DarkGreen");               // Specify a new Instance and override the Name             registry.AddInstanceOf<IWidget>().UsingConcreteType<AWidget>().WithName("MyInstance");                 registry.AddInstanceOf<IWidget>().UsingConcreteType<AWidget>();               // Specify a new Instance, override a dependency with a named instance             registry.AddInstanceOf<Rule>().UsingConcreteType<WidgetRule>().WithName("RuleThatUsesMyInstance")                 .UsingNamedInstanceFor<IWidget>("MyInstance");               // Specify a new Instance that specifies the concrete type used for a dependency             registry.AddInstanceOf<Rule>().UsingConcreteType<WidgetRule>().WithName("DarkGreenWidgetRule")                 .UsingConcreteType<AWidget>().ForDependency<IWidget>();               // Specify a new Instance, create an instance for a dependency on the fly             registry.AddInstanceOf<Rule>().UsingConcreteType<WidgetRule>().WithName("OrangeWidgetRule")                 .UsingDependency(                     Registry.Instance<IWidget>().UsingConcreteType<ColorWidget>()                         .WithProperty("Color").EqualTo("Orange"));               // Build an instance for IWidget, then setup StructureMap to return cloned instances of the             // "Prototype" (GoF pattern) whenever someone asks for IWidget named "Jeremy"             registry.AddInstanceOf<IWidget>().WithName("Jeremy").UsePrototype(new CloneableWidget("Jeremy"));               // Return the specific instance when an IWidget named "Julia" is requested             registry.AddInstanceOf<IWidget>( new CloneableWidget("Julia") ).WithName("Julia"); Adding PluginFamily's I put this off for years because I initially disliked the PicoContainer bootstrapping model, but people have asked for it.  In one line, here's a quick way to specify the default concrete type for a PluginFamily type.  Please note that this only works for classes that either have no constructor arguments or the types in the constructor argument list are all controlled by StructureMap.                 // Specify the default implementation for an interface                 registry.BuildInstancesOfType<IGateway>().WithDefaultConcreteType<StubbedGateway>(); Now, define the default instance for IWidget in code:                 // Define the default instance of IWidget                 registry.BuildInstancesOfType<IWidget>().AndTheDefaultIs(                     Registry.Instance<IWidget>()                         .UsingConcreteType<ColorWidget>()                         .WithProperty("Color").EqualTo("Red")                     ); Lastly I'm also going to add programmatic support for creating Profile's in memory as well to specify application "Modes," i.e. "Connected" or "Disconnected" or "Stubbed."  The programmatic DSL can be used with, or without, Xml or Attribute configuration.  When I make the release, I will write a tutorial on the new DSL grammar and might make a post on the mechanics of writing a Fluent Interface. Posted Feb 20 2007, 11:28 AM by Jeremy D. Miller with 4 comment(s) Filed under: StructureMap

• My Least Favorite Form of Requirements

  Just make it work like [system XYZ] does. Ok, and exactly what does [system XYZ] do?  Is that really what the client wants?  You know it's a little difficult to reproduce in 5 months what originally took 10-12 years don't you?  Maybe we could prioritize which features really do get ported? Two years ago I was part of a semi-successful rewrite effort of some truly heinous VB6 code.  The tribal knowledge was gone, and the code was extremely difficult to follow (hence the rewrite).  We ended up pulling most of the detailed requirements out by creating characterization tests by running inputs through the old code and setting up automated tests to ensure our new code did the exact same thing.  That's a double edged sword because the characterization tests aren't particularly readable or understandable -- and you potentially end up rewriting existing bugs! The absolute worst was a project I was on in 2003 where our mission was to "refactor" (we reused the graphics and threw away the rest, including the HTML) a purchased system until it was ready for production.  The business partner really wasn't interested in the project, so our total requirements were to make it work like XYZ.  The problem was that in this case XYZ was an unfinished prototype, so we basically made up the requirements the best that we could.  In the end, nobody walked away happy. Actually, come to think of it, there was a vague functional spec written very early on that nobody ever looked at again.   Posted Feb 20 2007, 07:33 AM by Jeremy D. Miller with 11 comment(s) Filed under: Ranting

• How do you extend and customize a database?

  Here's the scenario:  your application is frequently customized per customer, and that means the database has to accomodate new types of information.  My question to you is what do you think is the best (least bad) way?  Here are the strategems I've used or seen before: Allow custom fields to be added to the database tables.  Slippery slope to Hell in my experience.  As an internal IT architect, customizing 3rd party code filled me with dread because our upgrade path was shot and support was often harder.  Inside a product company that kind of customization adds maintainability costs, especially if you have to keep customer specific branches.  I have seen some systems accomodate this with custom data dictionaries and metadata driven architectures.  Needless to say, it's complex. Use wildcard fields at the end of tables like "string_field_1" or "int_field_1" that can be used for anything.  I refer to this kind of design as the "Mystery Meat" antipattern because it's never obvious what's in there. Store extended information in name/value extension tables.  Flexible, but often clumsy in execution.  More Mystery Meat.  Drives DBA's nearly insane.  Wait, that might be a plus. Serialize the extended properties as Xml or some other sort of Memento representation in a column.  Presupposes an Object Oriented Domain Model approach.  Requires some customization of the persistence layer, but I think some O/R tools support this (ActiveRecord in RoR does at least).  Doesn't work all that well for reporting needs.  Mystery Meat again. On a tangential note, is anybody using a true OODB?  Do you think it'll ever be feasible/practical/common?  Maybe the XML hybrid databases will end up being better. Now that Smalltalk and its variants seem to be making a bit of a comeback, I wonder it that ends up making OODB's practical if they can just use continuations to take quick snapshots of an object.  All the OODB's I've ever read about were Smalltalk. Posted Feb 19 2007, 09:51 AM by Jeremy D. Miller with 13 comment(s) Filed under: Database and Persistence

• StructureMap vNext, StoryTeller Alpha 2, and a vacation

  StructureMap  Just a quick note, and to make myself a deadline, the next version of StructureMap will be dropping by the end of February.  This version is about fullblown Generics support, ease of use, and flexible configuration.  The new features are: Full auto-wiring support for Generic template types with a caveat - IService<T> good.  IService<T, U> good.  IService<T,U,V> wacky exceptions.  I've got no answers on that one. Bug fixes Flexible configuration -- move the configuration somewhere else, use multiple configuration files, pull from the .Net App.config, or something else altogether Turn off configuration altogether and register dependencies programmatically with... A Fluent Interface DSL API for programmatic "wiring" as an alternative to xml configuration or using attributes.  I hope it turns out to be useful, but if nothing else it's an excuse to play with Fluent Interface semantics on something nontrivial. Flexible and more forgiving xml configuration.  I'm going to make the StructureMap.config file stop looking like a referential database.  There will be options to make the configuration more terse and require fewer repetitive data.  There will be a blog post later on applying the Don't Repeat Yourself principle to StructureMap in an attempt to make it easier to use The Generics support is already in the Subversion trunk if you're so inclined. StoryTeller There's at least a couple shops that are doing some alpha testing on StoryTeller for me and I'm getting some useful feedback coming back.  Telerik has generously given us a license for their UI controls to be used for StoryTeller.  I've done quite a bit of work on the UI since Alpha #1.  As soon as I can get the UI prettier and eliminate some severe flickering I will make a second Alpha release some time in March.     Vacation  I'm off this coming week, so you'll have to go somewhere else for unnecessarily long and opinionated blog posts.  I'm catching up on some long overdue family time on the family farm where the internet is still just a rumour;)  Posted Feb 10 2007, 01:14 PM by Jeremy D. Miller with 1 comment(s) Filed under: StoryTeller, StructureMap

• Ruby.Net Lives! Is RoR on the CLR coming too?

  Funny, I had a couple conversations about Ruby.Net this week and we all were disappointed because we thought it was dead. Ruby.Net Lives!  New beta release. From the release, We have just started work on getting Ruby on Rails to run on Ruby.NET and have started work on adding interoperability features to allow .NET programs written in other languages to conveniently use Ruby components and vice versa. We hope to include some of these features in the next public release. Posted Feb 08 2007, 08:37 PM by Jeremy D. Miller with 11 comment(s)

• Requirements Quote

  from Ron Jeffries on the Agile Testing board I think "requirements" always means approximately "what we think, right now, that we need".   One of my pet peeves working in development is dealing with the waterfall-ish idea that requirements and analysis is done just because you clicked print on a Word document.  Posted Feb 07 2007, 10:14 AM by Jeremy D. Miller with 4 comment(s)

• Composite keys are evil

  The title says it all.  There's a section in Eric Evans' DDD book where he spends about 3-5 pages saying that composite keys are bad and surrogate keys are good (without just coming out and saying that).  At the time I thought he was being exceptionally wordy, but it's an awfully good point to drive home.  Composite keys (multi column primary keys) make any kind of Object/Relational mapping and persistance in general harder.  Life is so much easier with surrogate keys.  You can always make unique constraints where it's necessary.  When I've used NHibernate against a legacy database, we just cut our losses and added autonumber columns with a unique constraint to act as the identifier for NHibernate and left the old key alone.  It worked pretty well, but I don't think those classes had any relationships, so I don't know how that would work.  I'd like to recommend to my current client to quietly ditch their homegrown Java persistence tool in favor of Hibernate, but putting an O/R mapper on a legacy database is challenging.  iBatis maybe, but I don't know much about it yet.   From anybody older than me out there, why were Composite keys so widely used in older databases?  Was there a real reason?  All I've ever seen from composite keys is pain.  Posted Feb 01 2007, 09:05 AM by Jeremy D. Miller with 28 comment(s)