Sponsored By Aspose - File Format APIs for .NET

Aspose are the market leader of .NET APIs for file business formats – natively work with DOCX, XLSX, PPT, PDF, MSG, MPP, images formats and many more!

Lipstick on a Pig

Posted by Jeremy Miller on February 24, 2007

I'm a card carrying member of the SOA skeptics club, but I'm going to push for a dose of SOA thinking to my current client.  I do understand the potential benefits of things like BAM, BPM, and flexible orchestration if the entire enterprise were really Service Oriented, but I generally think of SOA as a kind of fool's gold for ivory tower integration architects who desperately want to play God without ever getting their hands dirty (my advice for these folks is to go get a copy of Civ IV).  For the most part, the SOA initiatives I've seen have only led to extra complexity and reduced productivity in development without ever coming close to the orchestration nirvana envisioned by the architects. 

As I see it, the positive goal of a Service Oriented Architecture is to eliminate duplication of effort in software projects across the enterprise by reusing existing code assets and reducing the cost of changing the behavior of software to keep pace with evolving business needs.  As I've stated in other posts, it's usually easier and generally less risky to create new behavior by writing brand new code instead of breaking into existing code to make modifications.  Call me Pollyanna, but I firmly believe that those very goals can often be achieved at a lower cost by a relentless focus on creating well-factored code backed up by a healthy dollop of test and build automation (SOA certainly doesn't abrogate the need for good code behind the service points, but that's a gripe for another day).  Aw, but here's the rub:  the mass majority of the code out there in production is of dubious quality and the kind of test and build automation that enables change simply doesn't exist for most systems. 

So back to the usefulness and appropriateness of an SOA solution, there's a phrase I've often heard or used in regards to exposing services to a nasty legacy codebase to try to extend the usefulness of said legacy code that's particularly applicable to my current project — "Lipstick on a Pig." 

This piggie needs some lipstick

Here's the scenario, you're building a new client application or work process automation solution that needs to integrate or use an existing legacy server application with oodles of business logic and functionality.  What can you do?  I can think of three options off the top of my head:

  1. Write the new client as part of the legacy system.  In our case the legacy system is in a niche language, and they are committed (justifiably so IMHO) to producing all new clients in either WinForms or Swing.  This option is not on the table.
  2. Reproduce the necessary functionality in the new application.  This is the approach that we ultimately were forced to choose, but nobody is, or should be, happy with this approach.  Reproducing the existing functionality made the project take much longer, and add considerably more risk to the project, than was envisioned when the project was launched.  I say risk because we're still not exactly sure what the correct business logic should be because the original code is not entirely readable or documented.  We have also increased my client's maintenance costs considerably by duplicating functionality without retiring the existing legacy code.  Anytime this functionality needs to change, they'll have to change and test both sets of code.  Because all of the integration happens through a shared transactional database, the implicitly tight coupling between our new code and the existing code is not obvious.
  3. Find a way to use the existing legacy code in place as it is.  In effect, they could desperately use a service layer of some sort between their legacy code and newer client applications to avoid the duplication.  In terms of design, decoupling the legacy code enough to slide in a service layer is going to be difficult (I'm betting most of the functionality is in the 2-tier UI clients), but in the end I think it'll provide a heck of a lot more value to them.  Once the service layer is in place, new development work should get easier because they can just hook right up to the services with a well-defined contract.  Even better yet, the service layer should provide the ability to go to work on the service backend and start moving code over to a better factored codebase in a mainstream platform.  Putting lipstick on this pig will generate business value.

This piggie needs to go home

Some of my attitude on SOA originates from my days inside internal IT in a Fortune 100 manufacturing company that really is a quite logical fit for a rational SOA approach.  On a particular project, we were tasked with making significant improvements to the nasty, failure prone shipping system (as in "shuts down the factory lines when it goes down).  A senior enterprise architect was given the responsibility for determining the overall architectural changes.  His entire plan pretty well came down to replacing the existing integration points with webMethods.  The point of his strategy was to decouple the clients of the nasty legacy application from the details of the legacy application first, then make improvements to the server system at some later point.  He was perfectly honest in describing his plans as "Lipstick on a Pig."  I've always been primarily an applications guy, so I was flipping out that we were spending a lot of energy that wouldn't result in a single, observable improvement in terms of uptime, performance, functionality, or reduced support costs.  Worst was the fact that the business was told they were paying for overdue structural improvements to eliminate downtime incidents.  I even expressed an opinion that the new webMethods integration infrastructure would only add network chatter to an already chatty system and further degrade the already deficient performance.  To this day, I don't think anything far short of a full rewrite would have represented a significant improvement (and they did replace it after I was gone).  Lipstick wasn't going to help this pig at all.

 

And no, I don't necessarily think SOA is a bad thing, but done badly or unnecessarily, SOA is a blight upon the land.

 

 

About Jeremy Miller

Jeremy is the Chief Software Architect at Dovetail Software, the coolest ISV in Austin. Jeremy began his IT career writing "Shadow IT" applications to automate his engineering documentation, then wandered into software development because it looked like more fun. Jeremy is the author of the open source StructureMap tool for Dependency Injection with .Net, StoryTeller for supercharged acceptance testing in .Net, and one of the principal developers behind FubuMVC. Jeremy's thoughts on all things software can be found at The Shade Tree Developer at http://codebetter.com/jeremymiller.
This entry was posted in Legacy Code, Maintainability. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://www.backlinkschecker.ws/backlinks/buylinks.html Purchase High PR Backlink

    Wow, I found this topic particularly interesting.

  • http://www.e-Crescendo.com jdn

    Vikas:

    Unfortunately, I don’t think there’s a parallel to what I’m doing (at least not immediately). Also I’m not a big fan of fit, though (admittedly) probably because I don’t understand it well enough yet.

    But, I’ll take another look at it. Thanks.

  • http://codebetter.com/blogs/jeremy.miller jmiller

    cough, cough

    I’m picking up StoryTeller a little bit today, and much more in the weeks to come. I’m *just* about to get a StructureMap release out of the way first. I’m starting to get some patches coming in to incorporate as well.

    Jeremy

  • http://www.vikasnetdev.blogspot.com/ Vikas Kerni

    Hi JDN,

    If you are building a Business Intelligence Tool which is an interface for ETL process, you are in same situation like me. We have automated unit/integration tests which just check whether everything is fine with application. That means that there are no runtime errors because of some bad data/new table Schema. These came handy when we upgraded our application to Net 2.0. With generics, we got rid of all strong type collections (around 5000 LOC). With nunit/mstest tests, we were able to catch lot of problems. Automated unit tests helped us to isolate these problems otherwise we would have wasted lot of time in debugging and finding the problem area.

    I did some research how to use FIT for accuracy test and wrote the following blog

    http://vikasnetdev.blogspot.com/2006/06/applying-fit-requires-different.html

    Problem is that I don’t like Fitnese wiki interface. It will be very hard to convince Business Analysts to use that and organize Fit Tables.
    I am waiting for Jeremy’ Story Teller for next step in that direction.

  • http://blog.magenic.com/aarone Aaron Erickson

    Hmmm…..

    SQL unit tests. Ugh.

    More often than not, at least in the space of SQL unit tests, I find that set theory is much more useful than contrived sample techniques for validation purposes.

    One of the main reasons I like LINQ is that it syntactically eliminates the need for a lot of unit tests. The day I have to unit test the fact that a join works as coded is the day I change careers and start working as a dairy cow farmer.

  • http://www.e-Crescendo.com jdn

    I know when I’m done by looking at the actual data output. In fact, I’m required to produce a data analysis that proves that the actual data output is within tolerance.

    It’s not exactly ‘by hand’ (perl code does it), and it is tedious, but I still don’t see how a unit test is going to tell me that the data, across the universe, is going to fall within tolerance.

    In other words, since I have to do the data analysis anyway, and the data analysis tells me right quick when the code is wrong or when something else is amiss…..

  • http://www.e-Crescendo.com jdn

    Ayende:

    I need to know, across the universe, that I am getting exact matches 90% of the time, not that I am within 90% accuracy for a particular data point.

    I know how to do that by running automated data analysis on the actual output. I don’t see how any unit test is going to tell me that.

  • http://vikasnetdev.blogspot.com Vikas Kerni

    Rewriting a legacy code which has evolved over 10 years is not going to be cheap. It would require million dollars budget.
    So putting a lipstick on pig is not a bad first step. After that certain more steps are required like facial,manicure, …..

    http://vikasnetdev.blogspot.com/2006/06/eight-point-checks-for-servicing.html

  • http://codebetter.com/blogs/jeremy.miller jmiller

    Um, to make sure it actually works? Repeatedly? To know when you’re done?

    You’re right that you’re not really doing TDD as a design technique, just writing tests, but do you really want to have to do that kind of tedious testing by hand and eyeball?

  • http://www.e-Crescendo.com jdn

    I’d rather just write the code. I don’t see that the unit tests do a *single* thing for me. If I need to do data analysis across 200+ points across half a million rows, why don’t I just write the code? What will a unit test do for me here?

    What’s the gain?

  • http://codebetter.com/blogs/jeremy.miller jmiller

    Your “unit tests” would be much larger than you’d really prefer, but that seems like something that would be very useful and possible to drive through tests. Funny, I had this discussion with some colleagues on another project a couple months back on whether or not TDD is possible on an ETL project.

    You might want to look at using a FIT engine instead of xUnit for data intensive operations.

    To Ayende’s point, you just right a test that does the comparison and then check for some sort of allowable rate of discrepancy.

  • http://www.ayende.com/Blog Ayende Rahien

    AssertDateMatch(expected,actual, Accuracy.Precentage(90));

  • http://www.e-Crescendo.com jdn

    It’s a moderately large ETL project (50 GB) where the source schema has changed in such a way that it is known ahead of time that the end result will have 1-2% differences, though it is not known exactly how or why.

    What’s the unit test I can create using TDD that will be almost completely green but not quite (in fact, if our data analysis shows a 100% match across the entire universe comparing the old schema with the new, that’s a flag that the code doing the processing is suspect)?

  • http://www.codebetter.com/blogs/raymond.lewallen rlewallen

    I’m also carrying that SOA skeptic card Jeremy. It certainly has its place, and is actually a core design of what I’m involved with now, although I believe its being over-implemented. Too often I believe systems are designed around SOA, instead of thinking about how SOA can help a properly designed system. I love this quote I found a long time ago by Grady Booch:

    “…SOA is just one part of establishing an enterprise architecture, and those organizations who think that imposing an SOA alone will bring order out of chaos are sadly misguided. As I’ve said many times before and will say again, solid software engineering practices never go out of style (crisp abstractions, clear separation of concerns, balanced distribution of responsibilties) and while SOA supports such practices, SOA is not a sufficient architectural practice.”

  • http://codebetter.com/blogs/jeremy.miller jmiller

    jdn,

    “I currently work on a project that would be negatively impacted it it were either agile or SOA driven, and anyone with experience with the project would understand why, but you can’t even try to argue the point with those who insist that “But we’ve done it successfully ourselves, you just aren’t doing it right.”

    Without getting into NDA type violations, why not? Just curious.

  • http://www.e-Crescendo.com jdn

    I would put CI and iterative development first.

    I had an interesting exchange with someone whom I respect highly about TDD. He had some basic code in a test example (designed for explanatory purposes) in which he tested the contructor of an object. So, his code was something like “object o = new object(32); Assert.IsEqual(o.Value, 32)”. I asked him why he didn’t have additional tests along the lines of “int i = 0; Assert.IsNotNull(i); Assert.IsEqual(i, 0)”.

    Now, I was being obnoxious, but there was a point. I’m a code gen junkie, so if you can’t trust your constructors with your code gen, you might as well take up something else.

    I currently work on a project that would be negatively impacted it it were either agile or SOA driven, and anyone with experience with the project would understand why, but you can’t even try to argue the point with those who insist that “But we’ve done it successfully ourselves, you just aren’t doing it right.”

    Which is not to say I’m arguing against your successfull experiences. I’m not. But people who are SOA skeptics should also be cognizant that there are people who are agile skeptics, with equally good reason.

  • http://codebetter.com/blogs/jeremy.miller jmiller

    jdn,

    “Any experienced SOA person would probably say that you just haven’t had the right experience or been in an environment where ‘real’ SOA was being done”

    That thought had most certainly run through my mind as I was writing this post, and there is ‘something’ here. The comparison to “you just haven’t applied Agile correctly” is obvious. I suppose the lesson might be to actually linvest in learning how best to use a technique/practice/process before betting the farm on it. Or at least learning something about it before you blow it off.

    If you asked me what I think is the singlemost valuable practice in regards to successful software development, I would put forward TDD, with Continuous Integration & iterative development in general as close seconds. I had an exchange the other day with a fellow stating very directly that TDD had failed for him and everybody he had spoken with who had ever tried it out. We’re inevitably influenced by our experiences.

    I’m not skeptical about Agile development because I have 4 years worth of positive, practical experience with Agile practices. I think they make sense and I’ve seen superior technical products and smoother project progression compared to my experience with traditional waterfall development. Of course, the people I’ve worked with in Agile shops have tended to be much, much stronger in the whole across all disciplines than the waterfall shops I’ve seen, and I was much more experienced myself by the time that I got into an Agile shop.

    Thanks for commenting,

    Jeremy

  • http://www.e-Crescendo.com jdn

    As a general comment, I agree with you.

    As a more general comment, I find it…….ironic? No, that’s not the best word…..funny? No, that’s not it either…..hmmmm……I find it ‘something’ to hear an agile advocate be skeptical about SOA. Or just about anything else.

    It just seems, you know, ‘something.’ Any experienced SOA person would probably say that you just haven’t had the right experience or been in an environment where ‘real’ SOA was being done, that you in fact worked in a completely non-SOA environment, since everyone knows SOA != web services.

    Kind of like what I hear from agile people when I say I’ve never seen a successful agile implementation of anything, ever. That I think there’s ‘something good in it’ (which sounds condescending, but you know, the unit testing stuff, etc.), but most of the rest of it sounds like a bunch of crapola. “Well, you’ve never really seen real agile.”

    But like I said, I agree with you. I’m also an SOA skeptic. I just think it is ….. ‘something’ to hear it from you.

    jdn