I’m writing some short papers on development practices as part of preparing for DevTeach. One paper is a statement of my belief and philosophy of continuous design, and the other paper is an attempt to provide a justification for dumping classic functional specifications in favor of managing requirements with user stories and acceptance tests. Before I publish these pieces (they will be blog posts sometime soon), I thought it might be worthwhile to first talk about the value proposition of any software process or practice. In no small part, I want to defend the very idea of talking about software processes and practices as a valuable contribution to software development. Before I talk about the practices that I espouse, I want to boil things down to first causes to understand what I’m trying to accomplish by using these practices, processes, and philosophies.
I occasionally bump into the idea that there is a direct linear relationship between a software team’s throughput and the number of coders on the project. That just isn’t true because there are so many other variables at play. There’s no escaping the fast that one very large variable is the process the project team and organization follows and the practices that the team uses to build software. It’s fashionable to try to cut off arguments or discussions about software processes by saying it’s “not the methodology, it’s the man.” It’s even the very first “over” value judgement in the Agile Manifesto (emphasis mine):
Individuals and interactions over processes and tools
Yes, people are the first order variable in the productivity equation of any software project, but focus on the part of the manifesto that I bolded – “Interactions.” How team members interact with each other, how they communicate, when they communicate are all major contributors to a team’s success or failure. For that matter, what qualities makes a team member more or less productive? Skill, knowledge, and effort level are obviously big parts of the equation, but I’d also throw out the practices and discipline that an individual uses as well. Think of two developers of the exact same skill level. The first is diligent about unit testing, running the build locally, and checks code in often. The second developer codes for a week at a time, then starts to do adhoc testing and debugging until the code works. Who’s more effective? Betcha it’s the first developer. Moreover, the second developer can easily do harm by making the other developer’s jobs harder by causing merge difficulties and delaying testing.
It’s almost a matter of faith (and very consistent with my experience) that the difference in productivity between developers varies by an order of magnitude or more. As far as software developers goes, I’m an unabashed elitist. I grok design patterns and OOP, I can fly through ReSharper keyboard shortcuts, I can write testable code without slowing down. Can I code 10X faster than the average developer? More importantly, can I create production quality code that’s ready to ship and makes the customer happy 10X faster than the average developer? Maybe. Sometimes. When everything else around me allows me to. Specifically, I can create working, shippable code from design to a verified implementation at my maximum efficiency only with effective and timely assistance of other people like business analysts, testers, and even a good project manager. I’m especially effective inside a supportive environment that gives me the right information and feedback at the right time.
Think of all the scenarios that either slow down a software team or represent wasted effort by the team:
- Waiting for missing requirements
- Struggling through ambiguous requirements
- Not having access to the business experts
- Delays from handoff’s between disciplines
- Testers and developers disagreeing over who’s interpretation of a requirement is correct
- Testers receiving code and not knowing what or how to test the completed software
- Having completed code, but not being able to quickly deploy that code for testing
- Rework from developer’s misunderstanding requirements
- Rework from the customer changing the requirements
- Assumptions about requirements or architecture being overturned
- Elaborate documents that quickly become obsolete
- Code that was written “because we’ll need it later,” and wasn’t actually needed
Of course, the single biggest waste is a completed system or project that does not create enough business value to have justified the project. Even if a team follows the initial requirements to a tee, the project can still be a failure. I want to prevent, or at least minimize, the occurrences of the list above. To do that requires a level of cooperation and interaction with other people. That’s where a process and team practices step in.
What’s a Good Process and Good Practices?
In the last section I talked about the common causes of waste and inefficiency. Instead of thinking about process and practices as overhead that gets in the way of coding, let’s talk about what good processes and practices can do to maximize a team’s throughput and prevent or at least minimize the occurrences of the items in the “waste” list above. I’m an avowed Agilist, but more than try to sell Agile practices in this post, let’s talk about how to choose and apply processes in the most effective way. Even if you’re choosing XP or Scrum, there’s a lot of variation in how you apply Agile practices. Pay attention to these underlying goals and your Agile adoption will bear more fruit.
So, your ideal process and collection of practices should:
- Result in the right system – Duh. Nothing is worse than writing something and then finding out that it’s not what the client really wants or needs.
- Eliminate Risk – Some elements of a project are riskier than others. The biggest risk is building the wrong system altogether. If not properly handled, architectural risks can cause a project to fail altogether. A good process will actively manage risk to either eliminate the risk early or shut down a project that is likely to fail to recover resources for a more likely success.
- Manage the Iron Triangle – It’s not as clear cut or mathematically precise as Newtonian physics, but project management is always bound by the iron triangle (Jeff Atwood calls it the iron stool — read the comments, too) – time, scope, and resources (and sometimes quality, but sacrificing quality usually isn’t efficient). Generally, you have to lock 2 of the three and let the 3rd slide. A good process recognizes the real constraints and creates a rational approach to manage the sliding axis of the iron triangle. To run a smooth project, you’re going to have to a lot of give and take between scope and time to find a compromise that fits both the project team and the customer.
- Create the right information in a timely manner – Back to the idea of uber-developers being 10X more productive than lesser developers. When I’m sitting on my hands or just playing with the build while I wait for detailed analysis questions from somewhere else, I’m no faster than the lowest script kiddie developer. Even worse is code or test plans that was built speculatively without the right information and has to be tossed away.
- Communicate information effectively – Communication is one of the biggest challenges on any nontrivial project. A lot of knowledge is going to be created by various members of the team. A big part of succeeding and running smoothly is how effectively that very knowledge creation can be transmitted to the other team members.
- Minimize waste – What is waste? It’s code that isn’t necessary. It’s code that has to be changed afterwards because it’s not right. It’s churn between developers and testers. It’s screwing up a testing build and making the testers idle. Waste is anytime people on the project aren’t able to work because some computing resource is unavailable or a server is down. Good practices minimize the churn and keep code migrations clean to avoid downtime. It’s any superfluous code written for the project. Any code that doesn’t contribute to the immediate business requirements should be treated as wasted effort (I’m ).
- Promote an economy of effort – Any repeated effort like code deployments or database changes is a potential tax on the team. Good teams do not tolerate tedious mechanical work, they automate it.
- Optimize the Whole – Everybody’s goal on a software project should be the successful delivery of working code. Every activity needs to be geared towards optimizing that goal and never locally optimizing any one part of the project. The requirements need to be done in a way that is easily consumable by developers, testers, and project management. Coding needs to be done in a way that facilitates easier coding. Configuration management practices might add more mechanical work to developers, but make the project as a whole go faster. Every eye needs to be focused on shipping code and never on simply making an intermediate deliverable.
- Provide a level of predictability – One thing you definitely want in any software project is predictability. You need to know how the project is really going and what expectations are realistic.
- Transparency – What’s going on? How much is really getting done, and to what quality? What elements of the project plan need to be actively managed and changed. The project manager has an actual role on the project too, and his effectiveness is strongly related to how accurate he or she understands the current state and direction of the project.
and just to let my pro-Agile, pro-Lean bias shine out like a bonfire:
- Accommodate Change – It is almost unheard of for a project to end up exactly the way it was envisioned at the onset. The customer is going to change their mind about requirements as the project unfolds. You’ll learn more about the requirements as you proceed. Priorities between features will change during the project. Technical assumptions will be disproved. Estimates will inevitably turn out to be wrong. The
Why Work Iteratively
Just to be clear, when I say “iterative” development I’m using that as shorthand for iterative and incremental (IID). By this definition, a Spiral lifecycle does not qualify. For the purpose of this post I’m talking about all IID lifecycles including formal iterative approaches like the Rational Unified Process or Microsoft Solutions Framework as well as Agile processes (XP, Scrum, FDD, etc.). On my last client engagement my development team attempted to work formal iterations within a nominally waterfall shop. Iterative development does require a more disciplined approach in the small than a waterfall and certainly requires more communication within the project. The iterative style of work caused some real consternation. My client’s chief architect was especially dubious on the value of iterative development. For his benefit and mine for the next time I have to have this particular argument, here is a set of reasons for finally moving away from the waterfall to an organized iterative process:
- The Standish Chaos report on the causes of IT project failures. The famous report, originally published in 1994, found that projects using an organized IID process had a noticeably smaller failure rate than strict waterfall projects. Even flexible waterfall projects had a lower failure rate than strict waterfall projects. That research might not necessarily support Agile processes, but it should move any arguments around software processes to an examination of how best to conduct iterative projects. In other words, do anything but waterfall.
- An iterative process strives to create production quality code at the end of each iteration. That means that full integration happens much earlier in the project. The deployment scripts are executed much earlier. Testing is engaged much earlier in the project. The customer sees implemented features much earlier on the project. The end result is that project risk can be eliminated or exposed much earlier on a project.
- Feedback. An iterative process provides much more opportunity for adaptability and feedback throughout the project lifecycle. What’s going to be more likely to consistently succeed, a lifecycle that requires you to be right upfront, or a lifecycle that gives you more opportunities to both detect problems and make midstream corrections? Hint, it’s the iterative lifecycle.
- Communication. With a waterfall mentality, the analysts, developers, and testers largely work sequentially with far too much communication being done by documents. Most of the time the three groups are working on completely different things at any one time. At the worst end of the spectrum, the three groups may not even be on the project at the same time. That sequential cycle can kill effective communication and all but eliminates adaptability. In an iterative project the three groups are working much more closely together on the same small set of features. Face to face talks at the whiteboard or at least phone calls are a much richer medium for communication. It’s much easier to build a common understanding of the project’s requirements and common language when the testers and developers are working concurrently.
- Real transparency is much more accurate because you’re only tracking the completion of features to production quality. Shippable code is the only real measure of progress. It’s easy to get overconfident when intermediate deliverables are being produced on time, but no UML diagram or specification ever comes with a guarantee of appropriateness. Only working code that’s accepted by the customer can be considered to be a true validation.
- Flexible scheduling — while still delivering software. You strive to exit each iteration with shippable code. At the start of each iteration you’re afforded the opportunity to change business priorities without losing the work that’s already been done.
- Partial success versus total failure. If your working iteratively you have something ready to ship at the end of each iteration. Even if the project is behind schedule, you’ll be able to put something into production at the project end date. If you’re doing waterfall testing, you won’t have anything ready to deploy until the very end. Again, back to the iron triangle. With an iterative project working in smaller increments I think you have much more control over the iron triangle negotiations.
- Client engagement. On my last project the ultimate customer was, shall we say, a major pain in the ass? He got noticeably easier to deal with when were able to consistently deliver a new build every two weeks with new functionality to demonstrate the project process. Working features in the system are much more impressive than paper documents.
Be Your Own Methodologist
Growing up, my Dad was well stocked with platitudes and folk sayings that I’ll probably inflict upon my son as well. One of the most irritating, but appropriate to this discussion, was “if you’re not helping me, you’re hurting me.” We need to take that attitude towards our software process. Any piece of the process that doesn’t add any value, seems to be just an inconvenient checkbox, or just generally gets in the way needs to be jettisoned. By the same token we need to be looking for gaps and inefficiencies in the way the team interacts. Find the causes of inefficiency or friction in the project and address those causes as you go along. Don’t tolerate reoccurring inefficiencies and friction.
What I’m trying to say is that you need to be your own Methodologist. Kent Beck, Ken Schwaber, Ivar Jacobsen, and Watts Humphrey are not stakeholders on your project. By all means, look for inspiration from lots of external sources, but in the end the people best suited to design a team’s process is the people on the team. No two projects or teams are exactly alike. I walk into every project with a list of practices and philosophies mostly originating from the Agile side of things, but the details of those practices always vary between projects and even during projects. Do more than simply own your process, constantly reflect on your process as the project proceeds. Treat your the way that your team works together as something that can be advantageous to the team instead of just some motions you have to go through.
That’s nice for the perfect world Jeremy, but my team doesn’t have any say over our process or practices. I know that most of us just don’t have full control over how our teams work, but even if the overarching corporate process is bad and stifling, there might be smaller changes that you can effect to make your project more successful. If things aren’t the way you want them to be, either change your organization or change your organization.