Production Moves in Ceremonial Waterfall
Production moves in big IT waterfall land are living nightmares of pure desperation and panic just waiting for the other shoe to drop. We usually had most of the team work an entire weekend in shifts just to be on hand as code and data was manually migrated. It’s not unusual to see the cots come out in project warrooms. I don’t think that’s what the Godfather gang have in mind when they “take it to the mattresses,” and the mafia seemed to have more fun with it than we did. As I’ve mentioned before, teams often made last minute coding changes right before go live because integration problems weren’t discovered until making the migration and everybody was afraid to miss the installation window.
Shifting attention back to the shiny agile world of Test Driven Development and Continuous Integration, we made a final push into a production staging environment with our new product yesterday (with no known defects I might add). After yesterday’s move, I left work on time and took my toddler to the YMCA pool with nary a cell phone at hand because I was perfectly confident everything was fine (I’ll probably pay for that this morning).
On a previous project last year for a previous employer we made our second release one Tuesday morning. I was quite happily working on a user story in the mainline code branch for the following release when the project manager came up to tell us the new release had been put into production without a hitch. The developers had completely forgotten about the production move!
Where I Think Agile Development Helps
So why do I think the production moves went so much better with agile development than my earlier projects in waterfall world (to be fair, the earlier projects were COM based with a fair amount of middleware and the later projects were .NET systems)? The waterfall shop had a admirably organized (but radically inconsistent) ceremonial change management process, but pitiful build management. I think most of the reasons I list below are applicable for any disciplined iterative and incremental process (I’m thinking RUP here), but agile processes put more emphasis on early and frequent feedback over almost anything else.
- Earlier, more frequent, and repeatable testing. Test Driven Development is really a design technique as much as anything, but it certainly creates a large body of automated unit tests that provide a safety net to keep new bugs from emerging. Add any kind of automated acceptance or regression test suite and that safety net gets much tighter.
- Continuous Integration. CI can be thought of as a lot of trial runs at automated code installation. Much the same way that TDD forces you to write code that is testable, maintaining an automated CI build forces you (if your thinking cap is on straight) to think hard about making your application or service easier to deploy and version. Some people will excoriate me for this statement, but I think a NAnt build is the best documentation for system installation because it unambiguously specifies what has to happen for the application to function (and it tends to stay synchronized with reality easier than a document). I’d still supplement that with a Wiki page though.
- Defects can be fixed faster and stay fixed. CI allows a team to push code fixes to testing much faster without sacrificing environmental control. In ceremonial waterfall land, code pushes required filling out forms and convincing an external group to make my code move first, often up to a six hour process if other projects got there first. By automating builds under the control of the project team, I think the control over the environment is much better even without a controlling environment authority outside the development team, and certainly much faster. When you do write an automated test for a defect as you fix it, the defect stays fixed. Lost time due to regression failures can be mitigated by automated testing.
- Project teams are, or should be, more self-reliant and self-organizing. What I mean is that a project team should be able to carry out its own configuration management without waiting for a sluggish external group. In agile projects and organizations I’ve consistently had fewer dependencies on external IT service teams. I think that goes a long way towards reducing project friction. It does require carrying a broader skillset on a project team and people willing to step outside their normal skill silo. I think that might be one of the largest barriers to agile adoption in big shops. Flame away on point number 4, I probably deserve it.