What is Scrum?

Scrum is a process for the empirical control of software development.  It enables teams to deliver working software in an iterative manner in order to maximize business value.

Underlying Principles

One of the biggest influences on Scrum is the evolving scientific discipline known as complexity theory.  It is concerned with the behavior over time of certain kinds of complex systems.  According to Jonathan Rosenhead,

“The systems of interest are dynamic systems – systems capable of changing over time – and the concern is with the predictability of their behavior. Some systems, though they are constantly changing, do so in a completely regular manner. For definiteness, think of the solar system, or a clock pendulum. Other systems lack this stability: for example, the universe (if we are to believe the ‘big bang’ theory), or a bicyclist on an icy road. Unstable systems move further and further away from their starting conditions until/unless brought up short by some over-riding constraint – in the case of the bicyclist, impact with the road surface.”

He continues,

“Stable (defined) and unstable (random) systems are the extremes. What is novel is the concept of something in between – chaotic behavior. It refers to systems which display behavior which, though it has certain regularities, defies prediction. Think of the weather as we have known it. Despite immense efforts, success in predicting the weather has been quite limited, and forecasts get worse the further ahead they are pitched. And this is despite vast data banks available on previous experience. Every weather pattern, every cold front is different from all its predecessors.” [1]

The differences in each weather pattern are a result of the combination of minute, undetectable differences in the inputs to the weather pattern called noise.  According to complexity theory, noise is pervasive in all complex (non-defined) systems.  The boundary between stable and unstable systems is called the “edge of chaos.”

With this in mind, most traditional software development methodologies (i.e., waterfall) are defined processes.  They are based on the assumption that the software development process is a stable system.  This assumption means that every action, practice, and technique is simple and repeatable.  This is the white box approach, where every activity in the process is visible and subject to suboptimization.

The results are the huge methodologies present today.  These “methodologies” tell you exactly what to do in every situation.  Numerous documents are produced at various stages; you only need to look up what stage you are in to get the list of diagrams and written documents you need to produce to show progress.

To keep the resulting methodology’s documentation from ballooning out of control, things must be made a little more general.  This abstraction, however, removes the specificity needed to ensure a repeatable process.  Thus the conundrum: methodologies that are repeatable are specific to only a very narrow subset of situations whereas more general methodologies are not repeatable.  The cause of the conundrum is the noise.

In contrast, empirical processes do not specify software development activities in advance.  Instead, they treat the development process as a black box.  Development starts with a best estimate at what will work.  Then the process is regularly inspected at defined intervals (i.e., every day), assessed, and adjusted if necessary.  Instead of a well-defined process, teams use a well-documented knowledge repository showing how various tasks were done in the past.  The difference is this knowledge is used for guidance going forward, not for repeatability of past actions.

Why is software an empirical process, rather than a defined process?  These three theories explain:

1. Ziv's Uncertainty Principle in Software Engineering states that uncertainty is inherent and inevitable in software development processes and products.
2. Humphrey's Requirements Uncertainty Principle states that for a new software system, the requirements will not be completely known until after the users have used it.
3. Wegner's Lemma states that it is not possible to completely specify an interactive system.

Given that software development is an empirical process, how can we use this knowledge to improve it?  Work in new product development holds some guidelines.  According to Takeuchi and Nonaka in “The New New Product Development Game,” the following characteristics are necessary for working at the edge of chaos [2]:

• Built-in instability
• Self-organizing project teams
• Overlapping development phases
• “Multilearning”
• Subtle control
• Organizational transfer of learning

These characteristics allow a process to harness noise, not fight it, to generate results.  Scrum was created from the ground-up with these characteristics in mind.

How to Scrum

Image courtesy of Mike Cohn [3]

Product Backlog

It all starts with the Product Backlog.  The product backlog is a compilation of all the features the software program requires, including functionality, technical architecture, known bugs, etc.  Anyone can add anything to the Product Backlog; however only one person, the Product Owner, prioritizes everything.  Not everything can be “high” priority!

The Product Owner works with everyone involved to come up with estimates for the items in the Product Backlog.  The estimates include all architecture, design, development, testing, and documentation necessary.  These estimates are used to help the Product Owner prioritize.  For example, an unimportant feature (as determined by the Product Owner) with a high initial estimate will probably be moved lower on the Product Backlog.  Also, the estimates are not binding.  Work can take longer or be completed more quickly.  The more realistic the estimates are, the more likely that the work will be completed close to the estimated time.

Sprint

Once the Scrum team, Scrum Master, and Product Owner are comfortable with enough items on the Product Backlog, it is time to start a Sprint.  The Scrum team chooses the highest priority features from the Product Backlog, with input from the Product Owner, and makes this subset the Sprint Backlog.  This is the amount of work the Scrum team thinks it can finish in the Sprint.

The Scrum team is given full authority to do whatever it needs to do to complete the Backlog by the end of the Sprint within the project’s constraints on budget, schedule, delivered functionality, and delivered quality.  The Scrum team should be cross-functional, having all the skills necessary to complete the Sprint’s goals.  Generally the team size is seven plus or minus two [6].  Any more than this, and the group should be split to form two Scrum teams; any less and most of the group dynamic is lost.

Sprints are usually one month long (30 calendar days).  A Sprint begins with creating the Sprint Backlog.  The Scrum team expands the Backlog into tasks that are 4-16 hours in length.  These are sometimes called miniature milestones [7] or inch-pebbles [8].  The remaining time on the tasks is updated each day.  The Sprint ends with a demonstration of the system to all involved stakeholders (users, management, Product Owner, etc.) and a review meeting.  Both of these are usually held on the same day.

The most important thing about the Sprint is that no outside influence can interfere with the Scrum team until the end of the Sprint.  The team maintains equilibrium during each increment, insulated from outside disturbance. Increments are punctuated every thirty days so that the team and management can evaluate what should be done during the next increment; this decision is based on what the team has accomplished and what the environment dictates is the next most important thing to do.

Daily Scrum

Every day there is a short (15 minute) Daily Scrum meeting.  In this meeting three questions are answered by every team member.  The questions are:

1. What did you do since the last Scrum?
2. What are you doing until the next Scrum?
3. What prevented you from doing work?

The Scrum team reports on their progress to the Scrum Master.  Anyone else may attend the meeting, however they cannot participate.  Any discussions that come up regarding a feature’s design or help for a team member should be spun off into another meeting that occurs after the Daily Scrum.  The Daily Scrum should be held in the same place at the same time every day.

The first two questions allow the Scrum Master to monitor the process qualitatively.  He or she can address problems quickly, since there is daily visibility into the work being performed.

The third question gives the Scrum Master all the information they need to help the team work better.  Sometimes it is as simple as “free coffee.”  Sometimes it can be as complex as organizational change.  Regardless, it is the Scrum Master’s job to do everything in his or her power to remove any impediments to work.  This is how bottom-up reengineering occurs. [4]

Managing with Scrum

Managing a Sprint

There are two sources of data for managing a Scrum team.  The quantitative source is a sum of the time remaining on all backlog tasks.  The team should be updating their estimated time remaining on each task each day, and the sum of this time remaining is the estimated number of hours of work remaining for the Sprint.  Once a task is complete, the Scrum team enters 0 for the task estimate.  This allows the estimate to be updated daily, giving an accurate picture of the current Scrum team’s position.

The burndown chart is used to visually display the time remaining in the backlog tasks.

Image courtesy of Mike Cohn [3]

The chart will bounce up and down as new information is discovered each day, as shown above.  By calculating the current slope in the number of hours remaining, an end date can be estimated.  Note that the axes of the graph are days (x axis) and hours (y axis).

If the end date is significantly earlier than the Sprint completion date, then more work is added from the Product Backlog.  If the end date is significantly later than the Sprint completion date, then enough of the lowest priority tasks are removed from the Sprint backlog to meet the date.  It is extremely important that the Sprint completion date does not change!

The qualitative source of data is the Daily Scrum meeting.  If what goes on in the Daily Scrum does not support the burndown chart, something is wrong and must be investigated.  If the two support each other, then the picture is relatively accurate.

Managing Releases

The burndown chart can also be applied to Release Planning.  It usually takes several Sprints to get the software to a releasable state.  Using the Product Backlog, a burndown chart can be created that shows the estimated amount of time to release.  The axes for the Product burndown chart are months (x axis) and days (y axis).  The change in units of measurement from the Sprint burndown chart is intended to show the increased level of uncertainty associated with the longer-term estimate.

Summing the estimates for all features in the Product Backlog provides a very rough estimate of the final product completion date.  The Product Backlog is not used very often, except for short-term products where there are few releases. [4]

Why use Scrum?

Advantages

• Productivity increases
  • Some Scrum teams have recorded a 4x increase in productivity
  • Most improve productivity by 10-20% depending on management commitment
• Continuous improvement
  • Scrum enables continuous, rapid, bottom-up reengineering
• Leverages the chaos
  • The product becomes a series of manageable chunks
  • Progress is made, even when requirements are not stable
  • Everything is visible to everyone
  • Team communication improves
  • The team shares successes along the way and at the end
  • Customers see on-time delivery of increments
  • Customers obtain frequent feedback on how the product actually works
  • A relationship with the customer develops, trust builds, and knowledge grows
  • A culture is created where everyone expects the project to succeed

Disadvantages

• Requires hand-on management, but not micromanagement
  • Management must be willing to make changes to help Scrum teams succeed
  • Scrum requires constant monitoring both quantitatively and qualitatively
• Requires management to delegate decision-making authority to the Scrum team
  • Managers must let Scrum teams make their own decisions, even allowing them to fail if necessary
• Scrum is new and different
  • People are resistant to change
  • Some workers are not comfortable with the responsibility Scrum enables

Conclusion

Scrum is many things.  Scrum is a management process.  It can wrap various software development practices, such as Extreme Programming, as long as they support iterative development.  Scrum is also a process for continuous improvement.  Each day the highest priority impediment to work is identified and removed or mitigated.  And most importantly, Scrum accepts the empirical nature of software development as a given, working within those constraints to enable the team to deliver as much business value as possible as quickly as possible.

References

[1] - Rosenhead, Jonathan.  Complexity Theory and Management Practice.

[2] – Sutherland, Jeff.  Agile Project Management with SCRUM: Theory and Practice.

[3] – Cohn, Mike.  The Scrum Development Process.

[4] – Schwaber, Ken and Beedle, Mike.  Agile Software Development with Scrum. Available at Amazon.com

[5] - Heylighen F.  The Problem with Suboptimization.

[6] – Miller, George.  The Magical Number Seven, Plus or Minus Two.

[7] – McConnell, Steve. Rapid Development. Available at Amazon.com

[8] – Rothman, Johanna.  How to Use Inch-Pebbles When You Think You Can't.

For More Info

Schwaber, Ken.  Agile Project Management with Scrum.