Why is deployment so hard? Where is my Agile deployment?!
Chad Myers wrote about using XML configuration and stored procedures to monkey patch fixes in the production environment on the ALT.NET mailing list:
Talk about dysfunctional, but we’ve learned to use these XML files, sprocs, and other coding techniques to allow us to manage things without having to go through the whole Act of Congress thing to get something deployed.
Dysfunctional Chad? I would have chosen the phrase "eerily a linear archetype." Deployment is where, in a properly controlled IT environment, everything comes to a halt. What Chad describes is ironically desirable, but I think the pain is unnecessary. Rather the pain should be unnecessary.
Let me tell a horror story of my own. This should send shivers (of recognition) up and down the collective spine of just about every developer reading this post. Back in my classic ASP days (which I miss, but that's a different story) we had an application where when we were developing or wanted to make a change we'd just go to the UNC share, open the page(s), change the embedded SQL, code, etc. and CTRL+S. Boom! In production! Now cross your fingers and hope you didn't mess things up. Even worse, imagine having to do a database change
UPDATE [Claims] SET [DateOfInjury] = NULL
Oops... think I forgot a WHERE in there. And, as if this weren't enough, every developer had access to a) the production DB and b) the production code. Collective code ownership at it's absolute worst. Needless to say, I've felt the burn and I've come a long way since those days. Like I said, I would be surprised if people don't recognize this blunder, and if you don't you're probably smarter than me and should have gotten a good cringe out of this tale.
Still, that model was damn fast if way too loose, fast, and reckless. There has to be something in between. At minimum you need several "checkpoints" in the development process where you build and examine the entire system:
- Integration - Build the components of a system from the HEAD or a given TAG
- QA Test - Build a runtime image of the system (including database) from a given TAG
- Acceptance Test - Copy of the QA Test system image for use by a user population
- Production Staging - Copy of the Acceptance Test system image for final production
This is the state of the world. We have locks. We finally have parity with the cargo trade industries of Medieval Europe and Ancient China. Problem is, and back to Chad's comment, locks are slow.
What if everything were a mini-patch to a production image? The reason we use XML and SQL is they don't require compilation. The are tiny units where you can, if you need to, go into a production system, open the file, and paste the update. Of course, you'll want to test that it works in the staging environment, but it's a whole lot easier than dropping in an assembly with all it's type baggage. Of course an assembly change is just a delta in-and-of-itself, so it's not the main issue.
The biggest impediment to smooth deployment is people. There are too many schedules, agendas, and subjective opinions between development and runtime. We can reduce some time with automated acceptance/performance/regression/etc tests, but the biggest determining factor is the quality coming out of the product from the development team? If intensive developer testing is in place, changes can be made quicker and with confidence. A happy side-effect of this practice is that the quality coming out of development is dramatically higher. There's a recording of Kent Back making this point, and, speaking from experience, I believe it 100%.
If you're interested in easing some deployment pain you might try reading this paper from the good people at ThoughtWorks, The Deployment Pipeline, which describes the state of our art in summary at least. I've also found the Recipes for Continuous Database Integration book to have some useful tidbits for eliminating pain specifically around handling databases. It's not free, but it's cheap.
Beyond that, there aren't many answers. I fear that this post is much like the ending of your typical Discovery Channel show about the Loch Ness Monster: "Does Nessie Exist? We don't know, bye!"