Last week I did a talk at DevTeach called "Creating a Maintainable Software Ecosystem" (the slide deck is here). On one of my slides I had the following bullet point:
NEVER build and deploy from a developer environment
Even though it showed up on the slide I don't think I got around to explaining why I don't like creating deployment packages on a developer workstation. Yesterday someone pinged me wanting to understand why I say you shouldn't make official builds on a development workstation. Why should I make it a standard that we don't do builds on a developer workstation?
I'll start by telling some horror stories. 5 years ago I built my very last Windows DNA web application and left the company the following spring. At the time our standard practice was to compile VB6 DLL's on a development workstation and check the build products into source control. I would then write up instructions for our configuration management group to manually deploy the COM DLL's as part of any upcoming production push. It worked exactly as well as you'd think. Almost a year after I left the company and project I got a phone call from a former colleague. He needed to make some changes to one of the COM DLL's, but when he downloaded the code out of source control a class was missing and the project wouldn't compile. He was screwed. It was worse than he knows because I think I made some last minute code changes in the actual production push that never made it into source control.
On another project I was asked to redesign a problematic system. I asked for the source code for the existing system. Nobody knew where it was. The original developer was a victim of a recent "Reduction in Force" and wasn't around to help. Fortunately, someone thought to go check the original developer's old cubicle and found an ancient looking notebook computer in an unlocked drawer that had a copy of the code on its hard drive. We think it was the code anyway.
Sure, these are horror stories of incompetence, but if we'd had a policy forbidding production pushes originating from developer workstations and some level of repeatable builds directly from source control we could have headed off some serious problems.
First Causes
Before I even bother to talk about how and where I want the builds made, let's put out the end goals as I see them:
- Repeatable builds. If anything should go wrong, and it will, can I quickly reproduce the build?
- I want to know the provenance of a build. Where did it come from? What exact version of the code is in there? Is it free of foreign bodies of code? When it comes time to troubleshoot production issues, will I know what code to look at?
- I don't want to screw up the production deployment. Building is never as simple as just compiling, even for XCopy applications.
- I want to know that the exact version of the code being deployed is the version of the code signed off by my QA folks
If you build on the developer workstation, and especially if you build from your normal working directory, I think you run a significant of blowing all four points. If your building from your source code on your box there's nothing to guarantee that all the code you're compiling against is both in source control and the proper version of the code. It's just the code you happen to have on your workstation. I don't care how conscientious you and your team is, you will flub up with source control from time to time.
Build on the Build Server
My teams do official builds to testing and production on the build server as part of the CruiseControl.Net build. One way or another, I like to have the CruiseControl.Net build put together a deployable package on each and every successful build and stash it off to the side somewhere easy to get to. Ideally, I like to have the test deploys be something that can be manually triggered to deploy a given version in a single step by the testers themselves. Why is this important? Let's go back to our original list of first causes:
- Repeatable builds. CruiseControl.Net runs an automated script with no manual intervention. If the script works, I say we have repeatability.
- I want to know the provenance of a build. CruiseControl.Net is pulling the code directly out of the master source control repository. If there's anything missing, the build won't complete. Besides, we want the master source control repository to be the single authoritative source right?
- I don't want to screw up the production deployment. All build steps should be automated. Manual builds are time consuming, but more importantly, they can be error prone. For example, several of my projects will ratchet up the tracing levels or inject stubs in development environments, but I'll use the NAnt build to guarantee that the correct configuration is used in production. I want that configuration step to be automated.
- I want to know that the exact version of the code being deployed is the version of the code signed off by my QA folks. See the following sections:
Tracking Build Number with CruiseControl.Net and NAnt
One of the first targets I add to the NAnt script of a new project is versioning support. In the StoryTeller.build file for my StoryTeller project I have the following target:
<target name="version" description="mark AssemblyInfo builds with the build number">
<property name="assembly-version" value="${project.version}.0000" />
<if test="${property::exists('CCNetLabel')}"> <property name="assembly-version" value="${project.version}.${CCNetLabel}" /> </if>
<echo message="I'm running"/>
<echo file="version.xml" message="<version>${assembly-version}</version>" />
<echo message="Marking this build as version ${assembly-version}" /> <asminfo output="src/CommonAssemblyInfo.cs" language="CSharp">
...
<attributes>
<attribute type="AssemblyVersionAttribute" value="${assembly-version}" /> ...
</attributes>
...
</asminfo>
</target>
Pay attention to the bolded parts. When the NAnt script runs it will create a new file called CommonAssemblyInfo.cs that contains assembly attributes to identify the product name and build version. This file is linked to each and every binary in my solution. As long as this target runs before the compilation I can embed the contents of the "assembly-version" property into the version number of each binary. The "assembly-version" is created at runtime by appending the current "CCNetLabel" property to the project version. Assuming the build is successful, I get a batch of binaries that can be clearly traced to a known CruiseControl.Net build. The "CCNetLabel" is supplied to NAnt from CruiseControl.Net itself and wouldn't be available if you simply ran the StoryTeller.build file manually. In fact, when I run the NAnt build manually all the binaries will be versioned (as of this writing) as "0.8.0.0000." Seeing the four zeros at the end of the build tells me this is a local developer build. In a real project that tells me that these assemblies are not to be trusted because I don't know where this build came from.
Ok, that's nice. You know what CC.Net build the assemblies came from, but how can you know exactly which version of the code in source control made up the binaries? Simple, we just have CC.Net tag the source control with the build number after successful builds. Fortunately, the CC.Net + Subversion combo makes this very simple. Here's the snippet from the ccnet.config file for StoryTeller to do just this:
<project name="StoryTeller" queue="StoryTeller">
<workingDirectory>d:\work\StoryTeller</workingDirectory>
<modificationDelaySeconds>10</modificationDelaySeconds>
...
<sourcecontrol type="svn">
<trunkUrl>http://storyteller.tigris.org/svn/storyteller/trunk</trunkUrl>
<workingDirectory>d:\work\StoryTeller</workingDirectory>
<tagBaseUrl>http://storyteller.tigris.org/svn/storyteller/tags</tagBaseUrl>
<tagOnSuccess>true</tagOnSuccess>
</sourcecontrol>
...
</publishers>
</project>
After each successful build, CC.Net will issue a command to Subversion to tag the current revision of the repository trunk as the CCNet build number. At any time I can retrieve the exact version of the code matching the binaries by the tag number. It's now trivial to retroactively create a support branch from a given CC.Net build. Any fights over what code made it into the official build are relatively easy to solve by simply going to the repository browser.
Removing Project Friction
The previous part talked about the mechanics of the build, but now let's talk over the workflow. I want to be able to deploy code to testing and even production with minimal friction. With a team of any size and embedded testers, I could easily be pushing code to the testers 3-5 times or more a day on average. To do this effectively, I want to shut down the possibility of errors and miscommunication between myself and the testers. Specifically, I'm concerned about these issues:
- We screwed up the push. I hate getting bug reports that are caused by botched testing pushes. 100% automation of builds with environment tests baby!
- The testers tested the bug fix before the bug fix was deployed. Don't waste the testers time. When you fix bugs or new stories, you always tell the testers which build number contains the fixes and new features. On the other side, the testers have to be aware of the current build number. Make the build number obvious in some way. Either teach them to check the assembly versions themselves, put the build number in some kind of "About" screen, or put the build number into the title bar. However you do it, strive to cut down on miscommunication problems betwixt you and the tester. "I thought you had already pushed that code" or "oops, you don't have the right code yet" comments need to become a thing of the past.
- I want to do the push fast and get right back to work. By wanting to work iteratively in smaller steps I'm going to need to make far more pushes to testing. I want them to be automated to spare my time.
Closing Thoughts
I find it more than a little ironic that traditionalists like to label Extreme Programming as irresponsible hacking, but yet I've learned far more about better configuration management practices from my XP experience than I ever did in a traditional software shop that was desperately chasing higher CMM levels.
We all want to be safe, know where things are, and generally stay under control. I also want to go fast and get stuff done. I don't want laborious configuration management practices holding me back and retarding my progress. I never want the fear of the configuration management process to stop me from making the right technical decisions. For all of these reasons and more, I strongly prefer using project automation wherever possible in place of fancy manual work processes. The case in point for me was a situation last spring where we hacked up some code because it was just too freaking hard to go through the client's database change management process.
I think this issue is another point in favor of blurring the lines between project roles. In my old traditional environment we had a separate configuration management team and the developers had nothing to do with this process other than filling out paperwork. As I discussed in my horror stories, I was able to color within the lines of their official process and still screw up the configuration management royally. In a small Agile shop us developers were responsible for configuration management. We implemented Continuous Integration and automated builds ourselves. We actually had far, far better control over what code was where inside our environment because change management was baked into the day to day development.
The morale of the story. If you have a request for a blog topic, you're not in a hurry, and I already have the content sitting around, I will happily write up a blog post for you.
Posted
Thu, Dec 6 2007 8:56 AM
by
Jeremy D. Miller